Category Archives: Metrics

Lazy NRR is Not NRR. Accept No Imitations or Subtitutes.

The other day I was looking at an ARR bridge [1] with a young finance ace.  He made a few comments and concluded with, “and net revenue retention (NRR) is thus 112%, not bad.”

I thought, “Wait, stop!  You can’t calculate NRR from an ARR bridge [2].”  It’s a cohort-based measure.  You need to look at the year-ago cohort of customers, get their year-ago ARR, get that same group’s current ARR, and then divide the current ARR by the year-ago.

“Yes, you can,” he said.  “Just take starting ARR, add net expansion, and divide by starting ARR.  Voila.”

Expecto patronum!  Protect me from this dark magic.  I don’t know what that is, I thought, but that’s not NRR.

Then I stewed on it for a bit.  In some ways, we were both right.

  • Under the right circumstances, I think you can calculate NRR using an ARR bridge [3]. But the whole beauty of the metric is to float over that definitional swamp and just divide two numbers — so I inherently don’t want to.
  • My friend’s definition, one I suspect is common in finance whiz circles, was indeed one shortcut too short. But, under the right circumstances, you can improve it to work better in certain cases.

The Trouble with Churn Rates
For a long time, I’ve been skeptical of calculations related to churn rates.  While my primary problems with churn rates were in the denominator [4], there are also potential problems with the numerator [5].  Worse yet, once churn rates get polluted, all downstream metrics get polluted along with them – e.g., customer lifetime (LT), lifetime value (LTV), and ergo LTV/CAC.  Those are key metrics to measure the value of the installed base — but they rely on churn rates which are easily gamed and polluted.

What if there were a better way to measure the value of the installed base?

There is.  That’s why my SaaStr 2019 session title was Churn is Dead, Long Live Net Dollar Retention [6].  The beauty of NRR is that it tells you want you want to know – once you acquire customers, what happens to them? – and you don’t have to care which of four churn rates were used.  Or how churn ARR itself was defined.  Or if mistakes were made in tracking flows.

You just need to know two things:  ARR-now and ARR-then for “then” cohort of customers [7].

A Traditional ARR Bridge
To make our point, let’s review a traditional ARR bridge.

Nothing fancy here.  Starting ARR plus new ARR of two types:  new logo customers (aka, new logo ARR) and existing customers (aka, expansion ARR).  We could have broken churn ARR into two types as well (shrinkage and lost), but we didn’t need that breakout for this exercise.

Now, let’s add my four favorite rows to put beneath an ARR bridge [8]:

Here’s a description:

  • Net new ARR = New ARR – churn ARR. How much the water level increased in the SaaS leaky bucket of ARR.  Here in 1Q21, imagine we spent $2,250K in S&M in the prior quarter.  Our CAC ratio would be a healthy 1.0 on a new ARR basis, but a far less healthy 2.1 on a net new ARR basis.  That’s due to our quarterly churn of 8%, which when annualized to 32%, flies off the charts.
  • Expansion as a percent of new ARR = expansion ARR / new ARR. My sense is 30% is a magic number for an established growth-phase startup.  If you’re only at 10%, you’re likely missing the chance to expand your customers (which will also show up in NRR).  If you’re at 50%, I wonder why you can’t sell more new logo customers.  Has something changed in the market or the salesforce?
  • Net expansion = expansion ARR – churn ARR. Shows the net expansion or contraction of the customer base during the quarter.  How much of the bucket increase was due to existing (as opposed to new) customers?
  • Churn rate, quarterly. I included this primarily because it raises a point we’ll hit when discussing lazy NRR.  Many people calculate this as = churn ARR / starting ARR (quarter).  That’s what I call “simple quarterly,” and you’ll note that it’s always lower than just “quarterly,” which I define as = churn ARR / starting ARR (year) [9].  The trace-precedents arrows below show the difference.

Lazy NRR vs. Cohort-Based NRR
With that as a rather extensive warm-up, let’s discuss what I call lazy NRR.

Lazy NRR is calculated as described above = (starting ARR + net expansion) / starting ARR.  Lazy NRR is a quarterly expansion metric.

Let’s look at a detailed example to see what’s really being measured.

This example shows the difference between cohort-based NRR and Lazy NRR:

  • Cohort-based NRR, a year-over-year metric that shows expansion of the two year-ago customers (customers 1 and 2).  This is, in my book, “real NRR.”
  • Lazy NRR, simple quarterly, which compares net expansion within the current quarter to starting ARR for that quarter.

The point of the trace-precendents arrows shows you that while the result coincidentally might be similiar (and in this case it is not), that they are measuring two completely different things.

Let’s talk about the last row, lazy NRR, cohort-based approximation, which takes starting ARR from year-ago customers, then adds (all) net expansion over the year and divides by the year-ago starting ARR. The problem?  Customer 3.  They are not in the year-ago cohort, but contribute expansion to the numerator because, with only an ARR bridge, you can’t separate year-ago cohort net expansion from new-customer net expansion.  To do that, you’d need to have ARR by customer [10].

Lazy NRR is not NRR.  NRR is defined as snapshot- and cohort-based.  Accept no substitutes or imitations.  Always calculate NRR using snapshots and cohorts and you’ll never go wrong.

Layer Cakes Tell No Lies
While I’m usually quite comfortable with tables of numbers and generally prefer receiving them in board reports, this is one area where I love charts, such as this layer cake that stacks annual cohorts atop each other.  I like these layer cakes for several reasons:

  • They’re visual and show you what’s happening with annual cohorts.
  • Like snapshot- and cohort-based NRR, they leave little to no room for gaming.  (They’re even harder to survivor bias as you’d have to omit the prior-period ARR.)
  • Given my now-distant geophysics background, they sometimes remind me of sedimentary rock.  (Hopefully yours don’t look like that, as unmetamorphized, sedimentary rock represents an NRR of only 100%!)

The spreadsheet for this post is available here.

(The post was revised a few times after initial publication to fix mistakes and clarify points related to the cohort-based approximation.  In the end, the resultant confusion only convinced me more to only and always calcuate NRR using cohorts and snapshots.)

# # #

Notes
Edited 10/8/22 to replace screenshots and fix spreadsheet bug in prior version.

[1] Starting ARR + new ARR (from new logo and expansion) – churn ARR (from shrinkage and lost) = ending ARR

[2] I probably should have said “shouldn’t.”  Turns out, I think you can, but I know you shouldn’t.  We’ll elaborate on both in this post.

[3] Those conditions include a world where customers expand or contract only on an annual basis (as you are unable to exclude expansion or contraction from customers signed during the year since they’re not sepearated in an ARR bridge) and, of course, a clear and consistent definition of churn, playing fairly with no gaming designed understate churn or overstate expansion, and avoidance of mistakes in calculations.

[4] Churn rates based off the whole ARR pool can halve (or more than halve) those based on the available to renew (ATR) pool, for example if a company’s mean contract duration is 2 or 3 years.  ARR churn rates are probably better for financial calculations, but ATR churn rates are a better indicator of customer satisfaction

[5] Examples of potential problems, not all strictly related to calculation of churn ARR, but presented for convenience.

  • Expansion along the way. Consider a customer who buys 100-unit contract, expands to 140 the next quarter (without signing a new one-year agreement that extends the contract), and then at the annual renewal renews for 130.  The VP of CS wants to penalize the account’s CSM for 10 units of churn whereas the CFO wants to tell investors its 30 units of expansion.  Which is it?  Best answer IMHO is 40 units of expansion in the second quarter and 10 units of churn at the renewal, but I’ve seen people/systems that don’t do it that way.   NRR sees 130% rate regardless of how you count expansion and churn.
  • Potential offsets and the definition of customer – division 1 has 100 units and shrinks to 80 at renewal while a small 40-unit new project starts at division 2. Is that two customers, one with 20 units of churn and one new 40-unit customer or is it one customer with 20 units of expansion?  NRR sees either 80% rate or 120% rate as function of customer definition, but I’d hope the NRR framing would make you challenge yourself to ask:  was division 2 really a customer and ergo belong in the year-ago cohort?
  • Potential offsets and the definition of product – a customer has 100 units of product A, is unhappy, and shrinks to A to 60 units while buying your new product B for 40. Did any churn happen?  In most systems, the answer is no because churn is calculated at the account level.  Unless you’re also tracking product-level churn, you might have trouble seeing that your new product is simply being given away to placate customers unhappy with your first one.  NRR is inherently account-level and doesn’t solve this problem – unless you decide to calculate product-level NRR, to see which products are expanding and which are shrinking.
  • Adjustments.  International companies need to adjust ARR for fluctuations in exchange rates.  Some companies adjust ARR for bad debt or non-standard contracts.  Any and all of these adjustments complicate the calculation of churn ARR and churn rates.
  • Gaming.  Counting trials as new customers and new ARR, but excluding customers <$5K from churn ARR calculations (things won’t foot but few people check).  Renewing would-be churning customers at $1 for two years to delay count-based churn reporting (ARR churn rates and NRR will see through this).  Survivor biasing calculations by excluding discontinuing customers.  Deferring ARR churn by renewing would-be churning customers with net 360 payables and a handshake (e.g., side letter) to not collect unless thing XYZ can be addressed (NRR won’t see through this, but cash and revenue won’t align).

[6] Since I now work frequently with Europe as part of my EIR job with Balderton Capital, I increasingly say “NRR” instead of “NDR” (net dollar retention), because for many of the companies I work with it’s actually net Euro retention.  The intent of “dollar” was never to indicate a currency, but instead to say:  “ARR-based, not count-based.”  NRR accomplishes that.

[7] Some companies survivor bias their NRR calculation by using the now-value and then-value of the now cohort, eliminating discontinuing customers from the calculation.   Think:  of the mutual funds we didn’t shut down, the average annual return was 12%.

[8] If you download the spreadsheet and expand the data groups you can see some other interesting rows as well.

[9] The flaw in “simple quarterly” churn is that, in a world that assumes pure annual contracts, you’re including people who were not customers at the start of the year and ergo cannot possibly churn in the calculations.  While you use the same numerator in each case, you’re using an increasing denominator and with no valid reason for doing so.  See here for more.

[10] In which case you might as well calculate NRR as defined, using the current and year-ago snapshots.

 

Talking Burn: Appearance on the Metric Stack Podcast on Cash Conversion Score and Related Metrics

It was a combination of luck and foresight that I started talking with Allan Wille and Lauren Thibodeau about capital efficiency as a potential topic for their Metric Stack podcast many months ago.  Because now, as the episode is coming out, capital efficiency is the hot topic of the day.  Good luck (if not for a bad reason), but I’ll take it.

Here are some of the things we discussed on the podcast:

  • If you think of startups as organisms that convert venture capital (VC) into ARR, then we need some metric for how efficiently they do that.
  • Bessemer’s cash conversion score (CCS) is one such metric
  • I believe Bessemer defines CCS upside-down; I find it more intuitive to use capital consumed as the numerator and ARR (to show for it) in the denominator — as you would do with a CAC ratio.
  • Using my formula (= 1/CCS) for aggregate burn, here are some benchmarks showing the correlation between investment IRR and CCS within Bessemer’s portfolio
  • < 1 is amazing (i.e., burning <$50M to get to $50M in ARR)
  • 1-2 is good (i.e., burning $50M to $100M to get to $50M)
  • 2-4 is questionable (i.e., burning $100M to $200M to get to $50M)
  • 4+ is bad (i.e., burning $200M+ to get to $50M)
  • On IRR, Bessemer companies with a ratio of <1x had an IRR of 120%, 1-2 had an IRR of 80%, and 2-4 had an IRR of 40%.
  • At some point, I’d somewhat tongue-in-cheekily defined a metric called hype factor on the theory that startup organisms actually produced two things:  ARR and hype.
  • The impact of strategy pivots on overall capital efficiency, what that can mean for future funding, and how that sometimes leads to recapitalizations and pay-to-play financing rounds

The episode is available on AppleSpotify, and YouTube.  Enjoy it!  And watch that burn!

The Pipeline Progression Chart:  Why I Like It Better Than Just Tracking Rolling Four-Quarter Pipeline

When asked, “how is it going?” many companies will respond with something akin to, “things are looking strong, the pipeline is up to $50M.”

Not a bad statement, but certainly an imprecise one.  “Over what timeframe?” you might ask.  To which you’ll typically hear one of two answers

  • “Uh, that’s the whole thing.” I don’t love this answer as many companies –particularly the ones who answer with all-quarter pipeline — let junk opportunities get parked in the 5Q+ pipeline.  (You can fix this by including a timeframe as part of the definition of opportunity and ensuring you review the entire pipeline whenever you do a pipeline scrub.)
  • “That’s the rolling four-quarter (R4Q) pipeline.” I don’t love this answer either because, in my experience, companies who focus on R4Q pipeline as their top pipeline metric tend not to put enough emphasis on pipeline timing.  It’s too easy to say in January, “this year’s number is $20M and we’ve got $50M in the pipeline already (2.5x pipeline coverage) so we are golden.”  The problem, of course, is if 80% of that pipeline is backloaded into Q4, then while “the year may look great,” you’re going to need to survive three wasteland quarters to get there.  Even if that $40M Q4 pipeline were real, which it usually isn’t, most sales VPs won’t be around in October to close it.

I never look at rolling-four-quarter pipeline for the simple reason that I’ve never had a rolling-four-quarter sales target.  We have quarterly targets.  Instead of looking at R4Q  pipeline and hoping it’s well distributed (over time and across sellers), my philosophy is the opposite:

Let’s focus on ensuring we start every quarter with 3.0x pipeline coverage.  If we do that, the year takes care of itself, as does the year after that.

Once you accept this viewpoint, a few things happen:

  • Someone needs to start forecasting day-1 next-quarter pipeline coverage. What’s the point of focusing on next-quarter coverage if no one is tracking it and taking corrective actions as needed?  As mentioned, I think that person should be the CMO.
  • We need to start tracking the progression of the pipeline over time. This quarter’s starting pipeline is largely composed of last-quarter’s next-quarter pipeline and so on.  Since there are so many ebbs and flows in the pipeline the best way to track this is via periodic snapshots.

Towards that end, here’s a chart I find useful:

Let’s examine it.

  • Each row is a snapshot of the pipeline, broken down by quarter, taken on the first day of the quarter. (Some allow a week or two, for pipeline cleanup before snapshotting, which is fine.)
  • We’re tracking pipeline dollars, not opportunity count, which generally works better if you have a range of deal sizes and/or a multi-modal distribution of average sales prices. Doing so, however, can leave you overconfident if you create new opportunities with a high placeholder value.  (See this post for what to do about that.)
  • We show pipeline coverage in the block on the right. Most people want this-quarter coverage of around 3.0.  Targets for next-quarter and N+2 quarter are usually less well understood because many people don’t track them.  Coverage needed in the out quarters is a function of your sales cycle length, but the easiest thing is to just start tracking it so you get a sense for what out-quarter coverage normally is.  If you’re worried about that 1.6x next-quarter coverage shown on the 7/1 snapshot, read this post for ideas on how to generate pipeline in a hurry.
  • It’s good to carry at least one year’s prior snapshots so you can see historical progression.  Even more is better.
  • I’m assuming bigger deals and longer sales cycles (e.g., 6 to 12 months) so you will actually have material pipeline in the out-quarters.  For a velocity model with 25-day sales cycles, I’d take this template but just switch the whole things to months.

The most fun part of this chart is this you read it diagonally.  The $7.5M in starting this-quarter pipeline at the 7/1/21 snapshot is largely composed of the $6.5M in next-quarter pipeline at the 4/1/21 snapshot and the $3M in pipeline at the 1/1/21 snapshot.  You can kind of see the elephant go through the snake.

When you add this chart to your mix, you’re giving yourself an early warning system for pipeline shortages beyond simply forecasting starting next-quarter pipeline.  You should do this, particularly with big deals and long sales cycles, because one quarter’s notice is usually not enough time to fix the problem.  Yes, you can and should always try to mitigate problems (and never give-up saying, “looks like we’re going to hit the iceberg”), but if you give yourself more advance notice, you’ll give yourself more options and a better chance at reaching the goal:  starting every quarter with 3.0x coverage.

Add this slide to your QBR template now!

Crash Course in Customer Success SaaS Metrics: Appearance on the ChurnZero podcast.

Earlier this week I appeared on a webinar with You Mon Tsang, founder and CEO of ChurnZero, a SaaS application aimed at helping subscription businesses reduce churn.

In this post, I will share the video of event, provide a link to the slides, provide a link to the Q&A wrap-up they posted, embed the video below, embed the slides below that, and finally provide a quick summary below that.

Here’s the video:

Here’s a copy of the slides:

Here’s a quick list of the topics we discussed:

  • ARR and MRR, and when to use which
  • Logo retention rate, why a count-based rate works best when your customers are more or less “all the same” on deal size, and that you should use a dollar-based rate when they’re not.
  • Available-to-renew (ATR) logo retention rate, which factors in only those customers who had a chance to renew or not.  If you’re an ARR-based company but do multi-year contracts not every customer has the chance to get out every year.
  • Gross revenue retention rate, and why it’s gathering steam as an important metric.  (Sometimes great expansion is hiding major churn and just looking at churn before expansion will reveal that.)
  • Net revenue retention (NRR), aka net dollar retention (NDR) for those who work only in dollars, which is probably the hottest SaaS metrics after ARR and ARR growth.
  • Lifetime value (LTV), and its fairly severe limitations.  I gave a talk on this at SaaStr two years back.
  • Customer acquisition cost (CAC) and the CAC ratio.  How it differs for new customer and expansion ARR.
  • LTV/CAC ratio.  An attempt to measure what something costs against what it’s worth, but one that has generally failed and is now being replaced by NRR.
  •  Benchmarks for many of these metrics from the KeyBanc 2021 SaaS Survey.

Thanks to all those who attended and thanks to You Mon for inviting me and Cori for executing it so well.

The Sales/Marketing Expense Ratio

Question:  how much does a $15M SaaS company spend on sales and marketing as a percent of ARR?  Answer:  35% (with 45% and 15% as the top and bottom quartiles).

Charts like this, from OpenView’s 2021 Financial & Operating Benchmarks survey, help to answer questions like that all the time.

Good SaaS executives keep these metrics in mind, and you can get them from KeyBanc, RevOps Squared, OpenView, or for bigger/public companies, sites like Meritech Public Comps, Public Comps, or Clouded Judgement.

A great revops or FP&A person will give the answer from multiple sources and explain the differences among them.  Moreover, they’d observe that sales and marketing (S&M) expense really should vary with growth rate, and they’d know that KeyBanc tracks that:

So if that $15M SaaS company is growing at 25%, then median S&M spend is 20% of revenue, whereas if it’s growing at 70%, then median S&M spend bulks up to 46%.

But that’s all SaaS Metrics 101.  Today, I’d like to hop to the 201 level by introducing a simple that metric that can reveal a lot and on which few people focus:  the sales/marketing expense ratio, which just equals sales expense divided by marketing expense.

To introduce the idea — quick, tell me what’s happening at this company:

My take:

  • The company is high relative to the benchmark
  • The company is not making much progress towards the benchmark
  • Sales is getting less efficient while marketing is getting more efficient

This situation is very common.  Sometimes, it’s justified bottom-up — e.g., we’re building a partners function in sales that is only slowly becoming productive and we’ve upgraded both marketing leadership and the martech stack to improve marketing efficiency.

Normally, it’s not.  In fact, normally, there’s no justification whatsoever.  When you ask, you get, “well, that’s just how the budget process worked out, the real focus was on improving S&M and we did.  Next question, please.”

Yes, you did improve S&M, but you put the “S&M” improvement 100% on the back of marketing (in fact, 200%) and with no bottom-up justification for why sales needs to get more expensive while marketing is going to magically become more efficient.  This is a mistake.  The likely result is underfed sellers screaming for pipeline, forming an angry mob with dogs and torches headed to the CMO’s office.

Let me tell you what’s going on when this happens:

  • Your CRO is a better negotiator than your CMO.  They better be.  If they’re not, you have an additional problem.
  • Your CRO has more negotiating leverage than the CMO.  They are negotiating the company number directly with the CEO and indirectly with the board.  This is high-stakes, board-level poker.
  • There’s usually no broken-out benchmark, typically only a combined benchmark, and given the prior two points, the CRO is just fine with that.
  • It’s easy to think that hiring sellers “leads directly” to new ARR than investing in marketing.  Why?  Because in enterprise software the bookings capacity model is typically driven off the number of sellers.  Yes, this is intellectually lazy and only works on the margin, but deep down, it’s what a lot of CEOs and CFOs feel.

So the CMO gets asked to suck it up, the board doesn’t notice the problem, the CFO notices but doesn’t want to rock the boat, and the CEO is just happy to get the plan approved.

Hopefully the CRO has the decency to attend the CMO’s going-away party in the fall.  Because if this process repeats itself for even a few years, that’s how it’s going to end.

So how do we fix this?

1. Shine a light on the problem, by adding the sales/marketing ratio to the in-line metrics presented in the plan.

I prefer to show it this way, which makes it clear we used to spend $2 in sales for every $1 in marketing, but that has crept up to over $3.  Showing the metric gives people the chance to ask the all-important question:  why?

The other way to show this is via “sales composition,” i.e., sales as a percent of sales and marketing:

In this case, you can say that sales has risen from two-thirds to three-quarters of S&M expense, and again ask why.  I think the former presentation is more intuitive, but the advantage of this presentation is that KeyBanc benchmarks it in this form:

2. Shine a light on your inverted funnel model.  Sometimes you can squeeze marketing expense just on the people side, but the real way you usually cut to these targets is by making a series of seemingly innocuous assumptions in your funnel.  Consider:

Saying, we need to take MQL to SQL from 10% to 12%, SQL to SAL up from 65% to 70%, and SAL to close up from 15% to 20% all sounds pretty reasonable.  When you combine these effects, however, you’re saying that you’re going to cut the cost of generating an opportunity by more than a third, from $2700 to $1800.  That should get some attention — without any explanation other than the compound effect of small tweaks, it sounds like an Excel-induced hallucination to me.

3. Get the CRO on your side.  Make them understand that squeezing marketing too hard for purely top-down reasons increases their risk on the plan.  Get them to go to bat for you saying, “we need to ensure we feed the sellers enough pipeline.”  Most boards solve for growth with one eye on the CAC and not the opposite.

4. Get the CFO on your side.  In my experience, the hardest person to convince in these debates is the CEO, not the CFO.  Why?  Because the CEO is the one and only person who must negotiate the plan target with the CRO and that’s always something of a painful process.  So, if you get the CRO and CFO on your side, you will greatly increase your odds of getting the CEO to along with you.  You win the CFO over by emphasizing risk.  Think:  “we’ve (finally) got the CRO signed up for the number, but we’ve squeezed marketing too hard and that’s adding risk to the plan” and then say the magic words, “we don’t want to miss plan — do we, CFO?”  They never do.

Conclusion
In a world where sales has more political power, better negotiating skills, and more negotiating leverage than their marketing colleagues, the somewhat natural state of affairs is for this ratio to slowly increase over time.  The question is:  should it?  Everyone on the e-team needs to take accountability for thinking about that and ensuring the company gets the right, not just the easy, answer.  And the CMO has the unique responsibility of ensuring they do.