I was chatting with a fellow SaaS executive the other day and the conversation turned to churn and renewal rates. I asked how he calculated them and he said:
Well, we take every customer who was also a customer 12 months ago and then add up their ARR 12 months ago and add up their ARR today, and then divide today’s ARR by year-ago ARR to get an overall retention or expansion rate.
Well, that sounds dandy until you think for a minute about survivor bias, the often inadvertent logical error in analyzing data from only the survivors of a given experiment or situation. Survivor bias is subtle, but here are some common examples:
- I first encountered survivor bias in mutual funds when I realized that look-back studies of prior 5- or 10-year performance include only the funds still in existence today. If you eliminate my bogeys I’m actually an below-par golfer.
- My favorite example is during World War II, analysts examined the pattern of anti-aircraft fire on returning bombers and argued to strengthen them in the places that were most often hit. This was exactly wrong — the places where returning bombers were hit were already strong enough. You needed to reinforce them in the places that the downed bombers were hit.
- Business books are plagued with survivor bias. Among other classics both Good to Great and Built to Last fall victim to it.
So let’s turn back to churn rates. If you’re going to calculate an overall expansion or retention rate, which way should you approach it?
- Start with a list of customers today, look at their total ARR, and then go compare that to their ARR one year ago, or
- Start with a list of customers from one year ago and look at their ARR today.
Number 2 is the obvious answer. You should include the ARR from customers who choose to stop being customers in calculating an overall churn or expansion rate. Calculating it the first way can be misleading because you are looking at the ARR expansion only from customers who chose to continue being customers.
Let’s make this real via an example.
The ARR today is contained in the boxed area. The survivor bias question comes down to whether you include or exclude the orange rows from year-ago ARR. The difference can be profound. In this simple example, the survivor-biased expansion rate is a nice 111%. However, the non-biased rate is only 71% which will get you a quick “don’t let the door hit your ass on the way out” at most VCs. And while the example is contrived, the difference is simply one of calculation off identical data.
Do companies use survivor-biased calculations in real life? Let’s look at my post on the Hortonworks S-1 where I quote how they calculate their net expansion rate:
We calculate dollar-based net expansion rate as of a given date as the aggregate annualized subscription contract value as of that date from those customers that were also customers as of the date 12 months prior, divided by the aggregate annualized subscription contract value from all customers as of the date 12 months prior.
When I did my original post on this, I didn’t even catch it. But therein lies the subtle head of survivor bias.
# # #
Disclaimers:
- I have not tracked the Hortonworks in the meantime so I don’t know if they still report this metric, at what frequency, how they currently calculate it, etc.
- To the extent that “everyone calculates it this way” is true, then companies might report it this way for comparability, but people should be aware of the bias. One approach is to create a present back-looking and a past forward-looking metric and show both.
- See my FAQ for additional disclaimers, including that I am not a financial analyst and do not make recommendations on stocks.