Category Archives: Startups

SaaS Product Power Breakfast with Stephanie McReynolds on Category Creation

Please join us for our next episode of the SaaS Product Power Breakfast at 8am Pacific on 5/20/21 as we have a discussion with former Alation CMO Stephanie McReynolds on the topic of category creation and her learnings as she helped drive the creation of the data catalog category and establish Alation as the leader in it [1].

In addition to her gig at Alation, Stephanie’s had a great career at many leading and/or category-defining vendors including E.piphany, Business Objects, PeopleSoft, Oracle, Aster Data, ClearStory, and Trifacta.

Questions we’ll address include:

  • Does a vendor create a category or do market forces?
  • In creating a category do you lead with product or solution?
  • How do you know if you should try to create a category?
  • What role do industry analysts play in category creation?
  • What happens once you’ve successfully created a category?  What next?

This should be great session on a hot topic.  See you there.  And if you can’t make it, the session will be available in podcast form.  We think of our show, like Dr. Phil, as a podcast recorded before a live (Clubhouse) studio audience.

# # #

[1] I am an angel investor in and member of the board of directors at Alation.

Appearance on the Metrics That Measure Up Podcast

“Measure or measure not.  There is no try.”

— My response to being called the Yoda of SaaS metrics.

Just a quick post to highlight my recent appearance on the Metrics That Measure Up podcast, hosted by Ray Rike, founder and CEO of RevOps^2, a firm focused on SaaS metrics and benchmarking.

Ray’s a great guy, passionate about metrics, unafraid of diving into the details, and the producer of a great metrics-focused podcast that has featured many quality guests including Bryon Deeter, Tom Reilly, David Appel, Elay Cohen, Mark Petruzzi / Paul Melchiorre, Sally Duby, Amy Volas, and M.R. Rangaswami.

In the episode, Ray and I discuss:

  • Top SaaS metrics — e.g., annual recurring revenue (ARR), ARR growth, net dollar retention (NDR), net promoter score (NPS), employee NPS, and customer acquisition cost (CAC) ratio
  • How metrics vary with scale
  • Avoiding survivor bias, both in calculating churn rates and in comparisons to public comparison benchmarks (comps) [1]
  • How different metrics impact the enterprise value to revenue (EV/R) multiple — and a quick place to examine those correlations (i.e., the Meritech comps microsite).
  • Win rates and milestone vs. cohort analysis
  • Segmenting metrics, such as CAC and LTV/CAC, and looking at sales CAC vs. marketing CAC.
  • Blind adherence to metrics and benchmarks
  • Consumption-based pricing (aka, usage-based pricing)
  • Career advice for would-be founders

If you enjoy this episode I’m sure you’ll enjoy Ray’s whole podcast, which you can find here.

# # #

Notes

[1] Perhaps more availability bias (or, as Ray calls it, selection bias) than survivor bias, but either way, a bias to understand.

Navel Gazing, Market Research, and the Hypothesis File

Ask most startups about their go-to-market (GTM) these days and they’ll give you lots of numbers.  Funnel metrics.  MQLs, SQLs, demos, and associated funnel conversion rates.  Seen over time, cut by segment.  Win/loss rates and close rates as well, similarly sliced.  Maybe an ABM scorecard, if applicable.

Or maybe more financial metrics like customer acquisition cost (CAC) ratio, lifetime value (LTV) or net dollar retention (NDR) rate.  Maybe a Rule of 40 score to show how they’re balancing growth and profitability.

And then you’ll have a growth strategy conversation and you’ll hear things like:

  • People don’t know who we are
  • But the people who know us love us
  • We’re just not seeing enough deals
  • Actually, we are seeing enough deals, but we’re not making the short list enough
  • Or, we’re making the short list enough, but not winning enough.

And there are always reasons offered:

  • We’re not showing enough value
  • We’re not speaking to the economic buyer
  • We’re a vitamin, not a pain killer
  • We’re not aligned with their business priorities
  • People don’t know you can solve problem X with our solution
  • Prospects can’t see any differentiation among the offerings; we all sound the same [3]
  • They don’t see us as a leader
  • They don’t know they need one
  • They know they need one but need to finish higher priorities first

It’s an odd situation.  We are literally drowning in funnel data, but when it comes to actually understanding what’s happening, we know almost nothing.  Every one of the above explanatory assertions are assumptions.   They’re aggregated anecdotes [4].  The CRM system can tell us a lot about what happens to prospects once they’re in our funnel, but

  1. We’re navel gazing.  We’re only looking at that portion of the market we engaged with.  It’s humbling to take those assertions and mentally preface them with:  “In that slice of the market who found us and engaged with us, we see XYZ.”  We’re assuming our slice is representative.  If you’re a early-stage or mid-stage startup, there’s no reason to assume that.  It’s probably not.
  2. Quantitative funnel analysis is far better at telling you what happened than why it happened.  If only 8% of our stage 2 opportunities close within 6 quarters, well, that’s a fact [5].  But companies don’t even attempt to address most of the above explanatory assertions in their CRM, and even those times when they do (e.g., reason codes for lost deals), the data is, in my experience, usually junk [6].  And even on the rare occasion when it’s not junk, it’s still the salesrep’s opinion as to what happened and the salesrep is not exactly an unbiased observer [7].

What’s the fix here?  We need to go old school.  Let’s complement that wonderful data we have from the CRM with custom market research, that costs maybe $30K to $50K, and that we run maybe 1-2x/year and ideally right before our strategic planning process starts [8].  Better yet, as we go about our business, every time someone says something that sounds like a fact but is really an assumption, let’s put it into a “hypothesis file” that becomes a list of a questions that we want answered headed into our strategic and growth planning.

After all, market research can tell us:

  • If people are aware of us, but perhaps don’t pick us for the long list because they have a negative opinion of us
  • How many deals are happening per quarter and what percent of those deals we are in
  • Who the economic buyer is and ergo if we are speaking to them
  • What the economic buyer’s priorities are and if we are aligning to them
  • When features are most important to customers shopping in the category
  • What problems-to-be-solved (or use-cases) they associate with the category
  • Perceived differences among offerings in the category
  • Satisfaction with various offerings with the category
  • If and when they intend to purchase in the category
  • And much more

Net — I think companies should:

  • Keep instilling rigor and discipline around their pipeline and funnel
  • Complement that information with custom market research, run maybe 1-2x/year
  • Drive that research from a list of questions, captured as they appear in real time and prompted by observing that many of these assertions are hypotheses, not facts — and that we can and should test them with market research.

 

# # #

Notes

[1] As many people use “demo” as a sales process stage.  Not one I’m particularly fond of [2], I might add, but I do see a lot of companies using demo as an intermediate checkpoint between sales-accepted opportunity and closed deal — e.g., “our demo-to-close rate is X%”

[2] I’m not fond of using demo as a stage for two reasons:  it’s vendor-out, not customer-in and it assumes demo (or worse yet, a labor-intensive custom demo) is what’s required as proof for the customer when many alternatives may be what they want — e.g., a deep dive, customer references, etc.  The stage, looking outside-in, is typically where the customer is trying to answer either (a) can this solve my problem or (b) of those that can solve my problem is this the one I want to use?

[3] This is likely true, by the way.  In most markets, the products effectively all look the same to the buyer!  Marketing tries to accentuate differentiation and sales tries to make that accentuated differentiation relevant to the problem at hand, but my guess is more often than not product differentiation is the explanation for the selection, but not the actual driver — which might rather be things like safety / mistake aversion, desire to work with a particular vendor / relationship, word of mouth recommendations, belief that success is more likely with vendor X than vendor Y even if vendor X may (perhaps, for now) have an inferior product)

[4] As the saying goes, the plural of anecdote is not data.

[5] And a potentially meaningless one if you don’t have good discipline around stages and pipeline.

[6] I don’t want to be defeatist here, but most startups barely have their act together on defining and enforcing / scrubbing basics like stages and close dates.  Few have well thought-out reason codes.

[7] If one is the loneliest number, salespersonship is the loneliest loss reason code.

[8] The biggest overlooked secret in making market research relevant to your organization — by acting on it — is strategically timing its arrival.  For example, win/loss reports that arrive just in time for a QBR are way more relevant than those that arrive off-operational-cycle.

A Ten-Point Sales Management Framework for Enterprise SaaS Startups

In this post, I’ll present what I view as the minimum sales management framework for an enterprise SaaS startup — i.e., the basics you should have covered as you seek to build and scale your sales organization [1].

  1. Weekly sheet
  2. Pipeline management rules, with an optional stage matrix
  3. Forecasting rules
  4. Weekly forecast calls
  5. Thrice-quarterly pipeline scrubs
  6. Deal reviews
  7. Hiring profiles
  8. Onboarding program
  9. Quarterly metrics
  10. Gong

Weekly Sheet
A weekly sheet, such as the one used here, that allows you to track, communicate, and intelligently converse about the forecast and its evolution.  Note this is the sheet I’d use for the CEO’s weekly staff meeting.  The CRO will have their own, different one for the sales team’s weekly forecast call.

Pipeline Management Rules with Optional Stage Matrix
This is a 2-3 page document that defines a sales opportunity and the key fields associated with one, including:

  • Close date (e.g., natural vs. pulled-forward)
  • Value (e.g., socialized, placeholder, aspiration, upside)
  • Stage (e.g., solution fit, deep dive, demo, vendor of choice)
  • Forecast category (e.g., upside, forecast, commit)

Without these definitions in place and actively enforced, all the numbers in the weekly sheet are gobbledygook.  Some sales managers additionally create a one-page stage matrix that typically has the following rows:

  • Stage name (I like including numbers in stage names to accelerate conversations, e.g., s2+ pipeline or s4 conversion rate)
  • Definition
  • Mandatory actions (i.e., you can be fired for not doing these)
  • Recommended actions (i.e., to win deals we think you should be doing these)
  • Exit criteria

If your stage definitions are sufficiently simple and clear you may not need a stage matrix.  If you choose to create one, avoid these traps:  not enforcing mandatory actions (just downgrade them to recommended) and multiple and/or confusing exit criteria.  I’ve seen stage matrices where you could win the deal before completing all six of the stage-three exit criteria!

Forecasting Rules
A one-page document that defines how the company expects reps to forecast.  For example, I’d include:

  • Confidence level (i.e., the percent of the time you are expected to hit your forecast)
  • Cut rules (e.g., if you cut your forecast, cut it enough so the next move is up — aka, the always-be-upsloping rule.)
  • Timing rules (e.g., if you can forecast next-quarter deals in this quarter’s forecast)
  • Management rules (e.g., whether managers should bludgeon reps into increasing their forecast)

Weekly Forecast Calls
A weekly call with the salesreps to discuss their forecasts.  Much to my horror, I often need to remind sales managers that these calls should be focused on the numbers — because many salespeople seem to love to talk about everything but.

For accountability reasons, I like people saying things that are already in Salesforce and that I could theoretically just read myself.  Thus, I think these calls should sound like:

Manager:  Kelly, what are you calling for the quarter?
Kelly:  $450K
Manager:  What’s that composed of?
Kelly:  Three deals.  A at $150K, B at $200K, and C at $100K.
Manager:  Do you have any upside?
Kelly:  $150K.  I might be able to pull deal D forward.

I dislike storytelling on forecast calls (e.g., stories about what happened at the account last week).  If you want to focus on how to win a given deal, let’s do that in a deal review.  If we want to examine the state of a rep’s pipeline, let’s do that in a pipeline scrub.  On a forecast call, let’s forecast.

I cannot overstate the importance of separating these three types of meetings. Pipeline scrubs are about scrubbing, deal reviews are about winning, and forecast calls are about forecasting.  Blend them at your peril.

Thrice-Quarterly Pipeline Scrubs
A call focused solely on reviewing all the opportunities in the sales pipeline.  The focus should be on verification:

  • Are all the opportunities actually valid in accordance with our definition of a sales opportunity?
  • Are the four key fields (close date, value, stage, forecast category) properly and accurately completed?
  • All means all.  While we can put more focus on this-quarter and next-quarter pipeline, we need to review the entire thing to ensure that reps aren’t dumping losses in out-quarters or using fake oppties to squat on accountants.

I like when these calls are done in small groups (e.g., regions) with each rep taking their turn in the hot seat.  Too large a group wastes everyone’s time.  Too small forgoes a learning opportunity, where reps can learn by watching the scrubs of other reps.

As a non-believer in alleged continuous scrubbing, I like doing these scrubs in weeks 2, 5, and 8 so the data presented to the executive staff is clean in weeks 3, 6, and 9.  See this threepart series for more.

Deal Reviews
As a huge fan of Selling Through Curiosity, I believe a salesperson’s job is to ask great questions that both reveal what’s happening in the account and lead the customer in our direction.  Accordingly, I believe that a sales manager’s job is to ask great questions that help salesreps win deals.  That is the role of deal review.

A deal review is a separate meeting from a pipeline scrub or a forecast call, and focused on one thing:  winning.  What do we need to learn or do to win a given deal?  As such,

  • It’s a typically a two-hour meeting
  • Run by sales management, but in a peer-to-peer format (meaning multiple reps attend and reps ask each other questions)
  • Where a handful of reps volunteer to present their deals and be questioned about them
  • And the focus is on asking reps (open-ended) questions that will help them win their deals

Examples:

  • What questions can you ask that will reveal more about the evaluation process?
  • Why do you think we are vendor of choice?
  • What are the top reasons the customer wouldn’t select us and how are we proactively addressing them?
  • How would we know if we were actually in first place in the evaluation process?

Hiring Profiles
A key part of building an enterprise SaaS company is proving the repeatability of your sales process.  While I have also written a threepost series on that topic, the TLDR summary is that proving repeatability begins with answering this question:

Can you hire a standard rep and onboard them in a standard way to reliably produce a standard result?

The first step is defining a hiring profile, a one-page document that outlines what we’re looking for when we hire new salesreps.  While I like this expressed in a specific form, the key points are that:

  • It’s specific and clear — so we can know when we’ve found one and can tell recruiters if they’re producing pears when we asked for apples.
  • There’s a big enough “TAM” so we can scale — e.g., if the ideal salesrep worked at some niche firm that only had 10 salespeople, then we’re going to have trouble scaling our organization.

Onboarding Program
The second key element of repeatability is onboarding.  Startups should invest early in building and refining a standard onboarding program that ideally includes:

  • Pre-work (e.g., a reading list, videos)
  • Class time (e.g., a 3-5 day live program with a mix of speakers)
  • Homework (e.g., exercises to reinforce learnings)
  • Assessment (e.g., a final exam, group exercise)
  • Mentoring (e.g., an assigned mentor for 3-6 months)
  • Reinforcement (e.g., quarterly update training)

In determining whether all this demonstrates a standard result, this chart can be helpful.

Quarterly Metrics
Like all functions, sales should participate in an estaff-level quarterly business review (QBR), presenting an update with a high-quality metrics section, presented in a consistent format.  Those metrics should typically include:

  • Performance by segment (e.g., region, market)
  • Average sales cycle (ASC) and average sales price (ASP) analysis
  • Pipeline conversion analysis, by segment
  • Next-quarter pipeline analysis, by segment
  • Customer expansion analysis
  • Win/loss analysis off the CRM system, often complemented by a separate quarterly third-party study of won and lost deals
  • Rep ramping and productivity-capacity analysis (e.g., RREs)

Gong
As someone who prides himself on never giving blanket advice: everybody should use Gong.

I think it’s an effective and surprisingly broad tool that helps companies in ways both tactical and strategic from note-taking to coaching to messaging to sales enablement to alerting to management to forecasting to generally just connecting the executive staff to what actually happens in the trenches — Gong is an amazing tool that I think can benefit literally every SaaS sales organization.

# # #

Notes
[1] This post assumes the existence of functioning upstream work and processes, including (a) an agreement about goals for percentage of pipeline from the four pipeline sources (marketing, SDR/out, sales/out, and partners), (b) a philosophically aligned marketing department, (c) good marketing planning, such as the use of an inverted funnel model, (d) good sales planning, such as the use of a bookings capacity model, and (e) proper pipeline management as discussed in this threepart series.

Using To-Go Coverage to Better Understand Pipeline and Improve Forecasting

This is the second in a three-part series focused on forecasting and pipeline.  In part I, we examined triangulation forecasts with a detailed example.  In this, part II, we’ll discuss to-go pipeline coverage, specifically using it in conjunction with what we covered in part I.  In part III, we’ll look at this/next/all-quarter pipeline analysis as a simple way to see what’s happening overall with your pipeline.

Pipeline coverage is a simple enough notion:  take the pipeline in play and divide it by the target and get a coverage ratio.  Most folks say it should be around 3.0, which isn’t a bad rule of thumb.

Before diving in further, let’s quickly remind ourselves of the definition of pipeline:

Pipeline for a period is the sum of the value of all opportunities with a close date in that period.

This begs questions around definitions for opportunity, value, and close date which I won’t review here, but you can find discussed here.  The most common mistakes I see thinking about the pipeline are:

  • Turning 3.0x into a self-fulfilling prophecy by bludgeoning reps until they have 3.0x coverage, instead of using coverage as an unmanaged indicator
  • Not periodically scrubbing the pipeline according to a defined process and rules, deluding yourself into thinking “we’re always scrubbing the pipeline” (which usually means you never are).
  • Applying hidden filters to the pipeline, such as “oh, sorry, when we say pipeline around here we mean stage-4+ pipeline.”  Thus executives often don’t even understand what they’re analyzing and upstream stages turn into pipeline landfills full of junk opportunities that are left unmanaged.
  • Pausing sales hiring until the pipeline builds, effectively confusing cause and effect in how the pipeline gets built [1].
  • Creating opportunities with placeholder values that pollute the pipeline with fake news [1A], instead of creating them with $0 value until a salesrep socializes price with the customer [2].
  • Conflating milestone-based and cohort-based conversion rates in analyzing the pipeline.
  • Doing analysis primarily on either an annual or rolling four-quarter pipeline, instead of focusing first on this-quarter and next-quarter pipeline.
  • Judging the size of the all-quarter pipeline by looking at dollar value instead of opportunity count and the distribution of oppties across reps [2A].

In this post, I’ll discuss another common mistake, which is not analyzing pipeline on a to-go basis within a quarter.

The idea is simple:

  • Many folks run around thinking, “we need 3.0x pipeline coverage at all times!”  This is ambiguous and begs the questions “of what?” and “when?” [3]
  • With a bit more rigor you can get people thinking, “we need to start the quarter with 3.0x pipeline coverage” which is not a bad rule of thumb.
  • With even a bit more rigor that you can get people thinking, “at all times during the quarter I’d like to have 3.0x coverage of what I have left to sell to hit plan.” [4]

And that is the concept of to-go pipeline coverage [5].  Let’s look at the spreadsheet in the prior post with a new to-go coverage block and see what else we can glean.

Looking at this, I observe:

  • We started this quarter with $12,500 in pipeline and a pretty healthy 3.2x coverage ratio.
  • We started last quarter in a tighter position at 2.8x and we are running behind plan on the year [6].
  • We have been bleeding off pipeline faster than we have been closing business.  To-go coverage has dropped from 3.2x to 2.2x during the first 9 weeks of the quarter.  Not good.  [7]
  • I can easily reverse engineer that we’ve sold only $750K in New ARR to date [8], which is also not good.
  • There was a big drop in the pipeline in week 3 which makes me start to wonder what the gray shading means.

The gray shading is there to remind us that sales management is supposed to scrub the pipeline in weeks 2, 5, and 8 so that the pipeline data presented in weeks 3, 6, and 9 is scrubbed.  The benefits of this are:

  • It eliminates the “always scrubbing means never scrubbing” problem.
  • It draws a deadline for how long sales has to clean up after the end of a quarter:  the end of week 2.  That’s enough time to close out the quarter, take a few days rest, and then get back at it.
  • It provides a basis for snapshotting analytics.  Because pipeline conversion rates vary by week things can get confusing fast.  Thus, to keep it simple I base a lot of my pipeline metrics on week 3 snapshots (e.g., week 3 pipeline conversion rate) [9]
  • It provides an easy way to see if the scrub was actually done.  If the pipeline is flat in weeks 3, 6, and 9, I’m wondering if anyone is scrubbing anything.
  • It lets you see how dirty things got.  In this example, things were pretty dirty:  we bled off $3,275K in pipeline during the week 2 scrub which I would not be happy about.

Thus far, while this quarter is not looking good for SaaSCo, I can’t tell what happened to all that pipeline and what that means for the future.  That’s the subject of the last post in this three-part series.

A link to the spreadsheet I used in the example is here.

# # #

Notes

[1]  In enterprise SaaS at least, you should look at it the other way around:  you don’t build pipeline and then hire reps to sell it, you hire reps and then they build the pipeline, as the linked post discusses.

[1A]  The same is true of close dates.  For example, if you create opportunities with a close date that is 18+ months out, they can always be moved into the more current pipeline.  If you create them 9 months out and automatically assign a $150K value to each, you can end up with a lot air (or fake news/data) in your pipeline.

[2]  For benchmarking purposes, this creates the need for “implied pipeline” which replaces the $0 with a segment-appropriate average sales price (ASP) as most people tend to create oppties with placeholder values.  I’d rather see the “real” pipeline and then inflate it to “implied pipeline” — plus it’s hard to know if $150K is assigned to an oppty as a placeholder that hasn’t been changed or if that’s the real value assigned by the salesrep.

[2A] If you create oppties with a placeholder value then dollar pipeline is a proxy for the oppty count, but a far less intuitive one — e.g., how much dollar volume of pipeline can a rep handle?  Dunno.  How many oppties can they work on effectively at one time?  Maybe 15-20, tops.

[3] “Of what” meaning of what number?  If you’re looking at all-quarters pipeline you may have oppties that are 4, 6, or 8+ quarters out (depending on your rules) and you most certainly don’t have an operating plan number that you’re trying to cover, nor is coverage even meaningful so far in advance.  “When” means when in the quarter?  3.0x plan coverage makes sense on day 1; it makes no sense on day 50.

[4] As it turns out, 3.0x to-go coverage is likely an excessively high bar as you get further into the quarter.  For example, by week 12, the only deals still forecast within the quarter should be very high quality.  So the rule of thumb is always 3.0x, but you can and should watch how it evolves at your firm as you get close to quarter’s end.

[5]  In times when the forecast is materially different from the plan, separating the concepts of to-go to forecast and to-go to plan can be useful.  But, by default, to-go should mean to-go to plan.

[6] I know this from the extra columns presented in the screenshot from the same sheet in the previous post.  We started this quarter at 96% of the ARR plan and while the never explicitly lists our prior-quarter plan performance, it seems a safe guess.

[7]  If to-go coverage increases, we are closing business faster than we are losing it.  If to-go coverage decreases we are “losing” (broadly defined as slip, lost, no decision) business faster than we are closing it.  If the ratio remains constant we are closing business at the same ratio as we started the quarter at.

[8]  A good sheet will list this explicitly, but you can calculate it pretty fast.  If you have a pipeline of $7,000, a plan of $3,900, and coverage of 2.2x then:  7,000/2.2 (rounded) = 3,150 to go, with a plan of 3,900 means you have sold 750.

[9] An important metric that can be used as an additional triangulation forecast and is New ARR / Week 3 Pipeline.

 

Using Triangulation Forecasts For Improved Forecast Accuracy and Better Conversations

Ever been in this meeting?

CEO:  What’s the forecast?
CRO:  Same as before, $3,400K.
Director 1:  How do you feel about it?
CRO:  Good.
Director 2:  Where will we really land?
CRO:  $3,400K.  That’s why that’s the forecast.
Director 1:  But best case, where do we land?
CRO:  Best case, $3,800K.
Director 2:  How do you define best case?
CRO:  If the stars align.

Not very productive, is it?

I’ve already blogged about one way to solve this problem:  encouraging your CRO think probabilistically about the forecast.  But that’s a big ask.  It’s not easy to change how sales leaders think, and it’s not always the right time to ask.  So, somewhat independent of that, in this series I’ll introduce three concepts that help ensure that we have better conversations about the forecast and ultimately forecast better as a result:  triangulation forecasts, to-go pipeline coverage, and this/next/all-quarter pipeline analysis.  In this post, we’ll cover triangulation forecasts.

Triangulation Forecasts

The simplest way to have better conversations about the forecast is to have more than one forecast to discuss.  Towards that end, much as we might take three or four bearings to triangulate our position when we’re lost in the backcountry, let’s look at three or four bearings to triangulate our position on the new annual recurring revenue (ARR) forecast for the quarter.

In this example [1] we track the forecast and its evolution along with some important context such as the plan and our actuals from the previous and year-ago quarters.  We’ve placed the New ARR forecast in its leaky bucket context [2], in bold so it stands out.  Just scanning across the New ARR row, we can see a few things:

  • We sold $3,000K in New ARR last quarter, $2,850K last year, and the plan for this quarter is $3,900K.
  • The CRO is currently forecasting $3,400K, or 87% of the New ARR plan.  This is not great.
  • The CRO’s forecast has been on a steady decline since week 3, from its high of $3,800K.  This makes me nervous.
  • The CRO is likely pressuring the VP of Customer Success to cut the churn forecast to protect Net New ARR [3].
  • Our growth is well below planned growth of 37% and decelerating [4].

I’m always impressed with how much information you can extract from that top block alone if you’re used to looking at it.  But can we make it better?  Can we enable much more interesting conversations?  Yes.  Look at the second block, which includes four rows:

  • The sum of the sales reps’ forecasts [5]
  • The sum of the sales managers’ forecasts [6]
  • The stage-weighted expected value (EV) of the pipeline [7] [8]
  • The forecast category-weighted expected value of the pipeline [9]

Each of these tells you something different.

  • The rep-level forecast tells you what you’d sell if every rep hit their current forecast.  It tends to be optimistic, as reps tend to be optimistic.
  • The manager-level forecast tells you how much we’d sell if every CRO direct report hit their forecast.  This tends to be the most accurate [10] in my experience.
  • The stage-weighted expected value tells you the value of pipeline when weighted by probabilities assigned to each stage. A $1M pipeline consisting of 10 stage 2 $100K oppties has a much lower EV than a $1M pipeline with 10 stage 5 $100K oppties — even though they are both “$1M pipelines.”
  • The forecast category-weighted expected value tells you the value of pipeline when weighted by probabilities assigned to each forecast category, such as commit, forecast, or upside.

These triangulation forecasts provide different bearings that can help you understand your pipeline better, know where to focus your efforts, and improve the accuracy of predicting where you’ll land.

For example, if the rep- and manager-level forecasts are well below the CRO’s, it’s often because the CRO knows about some big deal they can pull forward to make up any gap.  Or, more sinisterly, because the CRO’s expense budget is automatically cut to preserve a target operating margin and thus they are choosing to be “upside down” rather face an immediate expense cut [11].

If the stage-weighted forecast is much lower than the others, it indicates that while we may have the right volume of pipeline that it’s not far enough along in its evolution, and ergo we should focus on velocity.

Now, looking at our sample data, let’s make some observations about the state of the quarter at SaaSCo.

  • The reps are calling $3,400K vs. a $3,900K plan and their aggregate forecast has been fairly consistently deteriorating.  Not good.
  • The managers, who we might notice called last quarter nearly perfectly ($2,975K vs. $3,000K) have pretty consistently been calling $3,000K, or $900K below plan.  Worrisome.
  • The stage-weighted EV was pessimistic last quarter ($2,500K vs. $3,000K) and may need updated probabilities.  That said, it’s been consistently predicting around $2,600K which, if it’s 20% low (like it was last quarter), it suggests a result of $3,240K [12].
  • The forecast category-weighted expected value, which was a perfect predictor last quarter, is calling $2,950K.  Note that it’s jumped up from earlier in the quarter, which we’ll get to later.

Just by these numbers, if I were running SaaSCo I’d be thinking that we’re going to land between $2,800K and $3,200K [13].  But remember our goal here:  to have better conversations about the forecast.  What questions might I ask the CRO looking at this data?

  • Why are you upside-down relative to your manager’s forecast?
  • In other quarters was the manager-level forecast the most accurate, and if so, why you are not heeding it better now?
  • Why is the stage-weighted forecast calling such a low number?
  • What’s happened since week 5 such that the reps have dropped their aggregate forecast by over $600K?
  • Why is the churn forecast going down?  Was it too high to begin with, are we getting positive information on deals, or are we pressuring Customer Success to help close the gap?
  • What big/lumpy deals are in these numbers that could lead to large positive or negative surprises?
  • Why has your forecast been moving so much across the quarter?  Just 5 weeks ago you were calling $3,800K and how you’re calling $3,400K and headed in the wrong direction?
  • Have you cut your forecast sufficiently to handle additional bad news, or should I expect it to go down again next week?
  • If so, why are you not following the fairly standard rule that when you must cut your forecast you cut it deeply enough so your next move is up?  You’ve broken that rule four times this quarter.

In our next post in the series we’ll discuss to-go pipeline coverage.  A link to the spreadsheet used to the example is here.

# # #

Notes

[1] This is the top of the weekly sheet I recommend CEOs to start their weekly staff meeting.

[2] A SaaS company is conceptualized as a leaky bucket of ARR.

[3] I cheated and look one row down to see the churn forecast was $500K in weeks 1-6 and only started coming down (i.e., improving) as the CRO continued to cut their New ARR forecast.  This makes me suspicious, particularly if the VP of Customer Success reports to the CRO.

[4] I cheated and looked one row up to see starting ARR growing at 58% which is not going to sustain if New ARR is only growing at ~20%.  I also had to calculate planned growth (3900/2850 = 1.37) as it’s not done for me on the sheet.

[5] Assumes a world where managers do not forecast for their reps and/or otherwise cajole reps into forecasting what the manager thinks is appropriate, instead preferring for managers to make their own forecast, loosely coupling rep-level and the manager-level forecasts.

[6]  Typically, the sum of the forecasts from the CRO’s direct reports.  An equally, if perhaps not more, interesting measure would be the sum of the first-line managers’ forecasts.

[7] Expected value is math-speak for probability * value.  For example, if we had one $100K oppty with a 20% close probability, then its expected value would be $100K * 0.2 = $20K.

[8] A stage-weighted expected value of the (current quarter) pipeline is calculated by summing the expected value of each opportunity in the pipeline, using probabilities assigned to each stage.  For example, if we had only three stages (e.g., prospect, short-list, and vendor of choice) and assigned a probability to each (e.g., 10%, 30%, 70%) and then multiplied the new ARR value of each oppty by its corresponding probability and summed them, then we would have the stage-weighted expected value of the pipeline.  Note that in a more advanced world those probabilities are week-specific (and, due to quarterly seasonality, maybe week-within-quarter specific) but we’ll ignore that here for now.  Typically, one way I sidestep some of that hassle is to focus my quarterly analytics by snapshotting week 3, creating in effect week 3 conversion rates which I know will work better earlier in the quarter than later.  In the real world, these are often eyeballed initially and then calculated from regressions later on — i.e., in the last 8 quarters, what % of week 3, stage 2 oppties closed?

[9]  The forecast category-weighted expected value of the pipeline is the same the stage-weighted one, except instead of using stage we use forecast category as the basis for the calculation.  For example, if we have forecast categories of upside, forecast, commit we might assign probabilities of 0.3, 0.7, and 0.9 to each oppty in that respective category.

[10] Sometimes embarrassingly so for the CRO whose forecast thus ends up a mathematical negative value-add!

[11] This is not a great practice IMHO and thus CEOs should not inadvertently incent inflated forecasts by hard-coding expense cuts to the forecast.

[12] The point being there are two ways to fix this problem.  One is to revise the probabilities via regression.  The other is to apply a correction factor to the calculated result.  (Methods with consistent errors are good predictors that are just miscalibrated.)

[13]  In what I’d consider a 80% confidence interval — i.e., 10% chance we’re below $2,800K and 10% chance we’re above $3,200K.