Category Archives: churn

Survivor Bias in Churn Calculations: Say It’s Not So!

I was chatting with a fellow SaaS executive the other day and the conversation turned to churn and renewal rates.  I asked how he calculated them and he said:

Well, we take every customer who was also a customer 12 months ago and then add up their ARR 12 months ago and add up their ARR today, and then divide today’s ARR by year-ago ARR to get an overall retention or expansion rate.

Well, that sounds dandy until you think for a minute about survivor bias, the often inadvertent logical error in analyzing data from only the survivors of a given experiment or situation.  Survivor bias is subtle, but here are some common examples:

  • I first encountered survivor bias in mutual funds when I realized that look-back studies of prior 5- or 10-year performance include only the funds still in existence today.  If you eliminate my bogeys I’m actually an below-par golfer.
  • My favorite example is during World War II, analysts examined the pattern of anti-aircraft fire on returning bombers and argued to strengthen them  in the places that were most often hit.  This was exactly wrong — the places where returning bombers were hit were already strong enough.  You needed to reinforce them in the places that the downed bombers were hit.

So let’s turn back to churn rates.  If you’re going to calculate an overall expansion or retention rate, which way should you approach it?

  1. Start with a list of customers today, look at their total ARR, and then go compare that to their ARR one year ago, or
  2. Start with a list of customers from one year ago and look at their ARR today.

Number 2 is the obvious answer.  You should include the ARR from customers who choose to stop being customers in calculating an overall churn or expansion rate.  Calculating it the first way can be misleading because you are looking at the ARR expansion only from customers who chose to continue being customers.

Let’s make this real via an example.

survivor bias

The ARR today is contained in the boxed area.  The survivor bias question comes down to whether you include or exclude the orange rows from year-ago ARR.  The difference can be profound.  In this simple example, the survivor-biased expansion rate is a nice 111%.  However, the non-biased rate is only 71% which will get you a quick “don’t let the door hit your ass on the way out” at most VCs.  And while the example is contrived, the difference is simply one of calculation off identical data.

Do companies use survivor-biased calculations in real life?  Let’s look at my post on the Hortonworks S-1 where I quote how they calculate their net expansion rate:

We calculate dollar-based net expansion rate as of a given date as the aggregate annualized subscription contract value as of that date from those customers that were also customers as of the date 12 months prior, divided by the aggregate annualized subscription contract value from all customers as of the date 12 months prior.

When I did my original post on this, I didn’t even catch it.  But therein lies the subtle head of survivor bias.

# # #

Disclaimers:

  • I have not tracked the Hortonworks in the meantime so I don’t know if they still report this metric, at what frequency, how they currently calculate it, etc.
  • To the extent that “everyone calculates it this way” is true, then companies might report it this way for comparability, but people should be aware of the bias.  One approach is to create a present back-looking and a past forward-looking metric and show both.
  • See my FAQ for additional disclaimers, including that I am not a financial analyst and do not make recommendations on stocks.

Churn:  Net-First or Sum-First?

****

Please note that this post has been superseded by A Fresh Look at How to Measure SaaS Churn Rates.  I’m leaving it posted to protect in-bound links only and to provide a referral to my latest material on this subject. 

****

 

While I’ve already done a comprehensive post on the subject of churn in SaaS companies and some perils in how companies analyze it, in talking with fellow SaaS metrics lovers of late, I’ve discovered a new problem that isn’t addressed by my posts.

The question?   When calculating churn, should you sum first (adding up all the shrinkage ARR) or net first (net shrinkage vs. expansion ARR and then sum that).  It seems like a simple question, but like so many subtitles in SaaS metrics, whether you net-first or sum-first, and how you report in so doing, can make a big difference in how you see the business through the numbers.

Let’s see an example.

net1

So what’s our churn rate:  a healthy -1% or a scary 15%?  The answer is both.  In my other post, I define about 5 churn rates, and when you sum first you get my “net ARR churn” rate [1], which comes in at a rather disturbing 15%.  When, however, you net first you end up a healthy -1% (“gross ARR churn”) rate because expansion ARR has more than offset shrinkage.  At my company we track both rates because each tells you a different story.

Thanks to the wonders of math, both the net-first and sum-first calculations take you to the same ending ARR number.  That’s not the problem.

The problem is that many companies report churn in a format not like my table above, but in something simpler like that looks like this below [2].

net2

As you can see, this net-first format doesn’t show expansion and shrinkage by customer.  I think this is dangerous because it can obscure real problems when shrinkage ARR is offset, or more than offset, by expansion ARR.

For example, customer 2 looks great in the second chart (“wow, $20K in negative churn!”).  In the first chart, however, you can see customer dropped 4 seats of product A and more than offset that by buying 8 seats of product B.  In fact, in the first chart, you can see that everyone is dropping product A and buying product B which is hidden in the second chart that neither breaks out shrinkage from expansion nor provides a comment as to what’s going on.  My advice is simple:  do sum-first churn and report both the “net ARR” and “gross ARR” renewal rates and you’ll get the whole picture.

Aside 1:  The Reclaimed ARR Issue
This debate prompted a second one with my Customers For Life (CFL) team who wanted to introduce a new metric called “reclaimed ARR,” the ARR that would have been lost on renewal but was saved by CFL through cross-sells, up-sells, and price increases.  Thus far, I’m not in love with the concept as it adds complexity, but I understand why they like it and you can see how I’d calculate it below.

net3

Aside 2:  Saved ARR
The first aside was prompted by the fact that CFL/renewals teams primarily play defense, not offense.  Like goalies on a hockey team, they get measured by a negative metric (i.e., the churn ARR that got away).   Even when they deliver offsetting expansion ARR, there is still some ARR that gets away, and a lot of their work (in the customer support and customer success parts of CFL) is not about offsetting-upsell, it’s about protecting the core of the renewal.  For that reason, so as to reflect that important work in our metrics, we’ve taken a lesson from baseball and the notion of a “save.”  Once the renewals come in, we add up all the ARR that came from customers who were, at any point in time since their last renewal, in our escalated accounts program and call that Saved ARR.    It’s best metric we’ve found thus far to reflect that important work.

# # #

[1] I have backed into the rather unfortunate position of using the word “net” in two different ways.  When I say “net ARR churn” I mean churn ARR net of (i.e., exclusive of) expansion ARR.  When I say net-first churn, I meant to net-out shrinkage vs. expansion first, before summing the customers to get total churn.

[2] Note that I properly inverted the sign because negative churn is good and positive churn is bad.