I was talking with my friend Tracy Eiler, author of Aligned to Achieve, the other day and she showed me a chart that they were using at InsideView to segment customers. The chart was a quadrant that mapped customers on two dimensions: renewal rate and retention rate. The idea was to use the chart to plot customers and then identify patterns (e.g., industries) so marketing could identify the best overall customers in terms of lifetime value as the mechanism for deciding marketing segmentation and targeting.
Here’s what it looked like:
While I think it’s a great chart, what really struck me was the thinking behind it and how that thinking reflects a dramatic evolution in the role of marketing across my career.
Back two decades ago when marketing was measured by leads, they focused on how to cost-effectively generate leads, looking at response rates for various campaigns.
Back a decade ago when marketing was measured by opportunities (or pipeline), they focused on how to cost-effectively generate opportunities, looking at response and opportunity conversion rates.
Today, as more and more marketers are measured by marketing-sourced New ARR, they are focused on cost-effectively generating not just opportunities, but opportunities-that-close, looking all the way through the funnel to close rates.
Tomorrow, as more marketers will be measured on the health of the overall ARR pool, they will be focused on cost-effectively generating not just opportunities-that-close but opportunities that turn into the best long-term customers. (This quadrant helps you do just that.)
As a company makes this progression, marketing becomes increasingly strategic, evolving in mentality with each step.
Starting with, “what sign will attract the most people?” (Including “Free Beer Here” which has been used at more than one conference.)
To “what messages aimed at which targets will attract the kind of people who end up evaluating?”
To “who are we really looking to sell to — which people end up buying the most and the most easily – and what messages aimed at which targets will attract them?”
To “what are the characteristics of our most successful customers and how can we find more people like them?”
The whole pattern reminds me of the famous Hubspot story where the marketing team was a key part forcing the company to focus on either “Owner Ollie” (the owner of a <10 person business) or “Manager Mary” (a marketer at a 10 to 1000 person business). For years they had been serving both masters poorly and by focusing on Manager Mary they were able to drive a huge increase in their numbers that enabled cost-effectively scaling the business and propelling them onto a successful IPO.
What kind of CMO does any CEO want on their team? That kind. The kind worried about the whole business and looking at it holistically and analytically.
I was chatting with a fellow SaaS executive the other day and the conversation turned to churn and renewal rates. I asked how he calculated them and he said:
Well, we take every customer who was also a customer 12 months ago and then add up their ARR 12 months ago and add up their ARR today, and then divide today’s ARR by year-ago ARR to get an overall retention or expansion rate.
Well, that sounds dandy until you think for a minute about survivor bias, the often inadvertent logical error in analyzing data from only the survivors of a given experiment or situation. Survivor bias is subtle, but here are some common examples:
I first encountered survivor bias in mutual funds when I realized that look-back studies of prior 5- or 10-year performance include only the funds still in existence today. If you eliminate my bogeys I’m actually an below-par golfer.
My favorite example is during World War II, analysts examined the pattern of anti-aircraft fire on returning bombers and argued to strengthen them in the places that were most often hit. This was exactly wrong — the places where returning bombers were hit were already strong enough. You needed to reinforce them in the places that the downed bombers were hit.
So let’s turn back to churn rates. If you’re going to calculate an overall expansion or retention rate, which way should you approach it?
Start with a list of customers today, look at their total ARR, and then go compare that to their ARR one year ago, or
Start with a list of customers from one year ago and look at their ARR today.
Number 2 is the obvious answer. You should include the ARR from customers who choose to stop being customers in calculating an overall churn or expansion rate. Calculating it the first way can be misleading because you are looking at the ARR expansion only from customers who chose to continue being customers.
Let’s make this real via an example.
The ARR today is contained in the boxed area. The survivor bias question comes down to whether you include or exclude the orange rows from year-ago ARR. The difference can be profound. In this simple example, the survivor-biased expansion rate is a nice 111%. However, the non-biased rate is only 71% which will get you a quick “don’t let the door hit your ass on the way out” at most VCs. And while the example is contrived, the difference is simply one of calculation off identical data.
Do companies use survivor-biased calculations in real life? Let’s look at my post on the Hortonworks S-1 where I quote how they calculate their net expansion rate:
We calculate dollar-based net expansion rate as of a given date as the aggregate annualized subscription contract value as of that date from those customers that were also customers as of the date 12 months prior, divided by the aggregate annualized subscription contract value from all customers as of the date 12 months prior.
When I did my original post on this, I didn’t even catch it. But therein lies the subtle head of survivor bias.
# # #
I have not tracked the Hortonworks in the meantime so I don’t know if they still report this metric, at what frequency, how they currently calculate it, etc.
To the extent that “everyone calculates it this way” is true, then companies might report it this way for comparability, but people should be aware of the bias. One approach is to create a present back-looking and a past forward-looking metric and show both.
See my FAQ for additional disclaimers, including that I am not a financial analyst and do not make recommendations on stocks.
I’m Dave Kellogg, advisor, director, consultant, angel investor, and blogger focused on enterprise software startups. I am an executive-in-residence (EIR) at Balderton Capital and principal of my own eponymous consulting business.
I bring an uncommon perspective to startup challenges having 10 years’ experience at each of the CEO, CMO, and independent director levels across 10+ companies ranging in size from zero to over $1B in revenues.
From 2012 to 2018, I was CEO of cloud EPM vendor Host Analytics, where we quintupled ARR while halving customer acquisition costs in a competitive market, ultimately selling the company in a private equity transaction.
Previously, I was SVP/GM of the $500M Service Cloud business at Salesforce; CEO of NoSQL database provider MarkLogic, which we grew from zero to $80M over 6 years; and CMO at Business Objects for nearly a decade as we grew from $30M to over $1B in revenues. I started my career in technical and product marketing positions at Ingres and Versant.
I love disruption, startups, and Silicon Valley and have had the pleasure of working in varied capacities with companies including Bluecore, Cyral, FloQast, GainSight, MongoDB, Recorded Future, and Tableau.
I previously sat on the boards of Granular (agtech, acquired by DuPont), Aster Data (big data, acquired by Teradata), and Nuxeo (content services, acquired by Hyland), and Profisee (MDM, exited to Pamlico).
I periodically speak to strategy and entrepreneurship classes at the Haas School of Business (UC Berkeley) and Hautes Études Commerciales de Paris (HEC).