Category Archives: Metrics

You Can’t Analyze Churn by Analyzing Churn

Posted on July 26, 2014 | 5 comments

One thing that amazes me is when I hear people talk about how they analyze churn in a cloud, software as a service (SaaS), or other recurring revenue business.

You hear things like:

“17% of our churn comes from emerging small business (ESB) segment, which is normal because small businesses are inherently unstable.”

“22% of our churn comes from companies in the $1B+ revenue range, indicating that we may have a problem meeting enterprise needs.”

“40% of the customers in the residential mortgage business churned, indicating there is something wrong our product for that vertical.”

There are three fallacies at work here.

The first is assumed causes. If you that 17% of your churn comes from the ESB segment, you know one and only one thing: that 17% of your churn comes from the ESB segment. Asserting small business stability as the cause is pure speculation. Maybe they did go out of business or get bought. Or maybe they didn’t like your product. Or maybe they did like your product, but decided it was overkill for their needs. If you want to how much of your churn came from a given segment, ask a finance person. If you want to know why a customer churned, ask them. Companies with relatively small customer bases can do it via a phone. Customers with big bases can use an online survey. It’s not hard. Use metrics to figure out where your churn comes from. Use surveys to figure out why.

The second is not looking at propensities and the broader customer base. If I said that 22% of your annual recurring revenue (ARR) comes from $1B+ companies, then you shouldn’t be surprised that 22% of your churn comes from them as well. If I said that 50% of your ARR comes from $1B+ companies (and they were your core target market), then you’d be thrilled that only 22% of your churn comes from them. The point isn’t how much of your churn comes from a given segment: it’s how much of your churn comes from a given segment relative to how much of your overall business comes from that segment. Put differently, what is the propensity of someone to churn in one segment versus another.

And you can’t perform that analysis without getting a full data set — of both customers who did churn and customers who didn’t. That’s why I say you can’t analyze churn by analyzing churn. Too many people, when tasked with churn analysis: say, “quick, get me a list of all the customers who churned in the past 6 months and we’ll look for patterns.” At that instant you are doomed. All you can do is decompose churn into buckets, but know nothing of propensities.

For example, if you noticed that in one country a surprising 89% of churn came from customers with blue eyes, you might be prompted to launch an immediate inquiry into how your product UI somehow fails for blue-eyed customers. Unless, of course, the country was Estonia where 89% of the population has blue eyes and ergo 89% of your customers do. Bucketing churn buys you nothing without knowing propensities.

The last is correlation vs. causation. Knowing that a large percentage of customers in the residential mortgage segment churned (or even have higher propensity to churn) doesn’t tell you why they are churning. Perhaps your product does lack functionality that is important in that segment. Or perhaps it’s 2008, the real estate crisis is in full bloom, and those customers aren’t buying anything from anybody. The root cause is the mortgage crisis, not your product. Yes, there is a high correlation between customers in that vertical and their churn rate. But the cause isn’t a poor product fit for that vertical, it’s that the vertical itself is imploding.

A better, and more fun, example comes from The Halo Effect, which tells the story that a famous statistician once showed a precise correlation between the increase in the number of Baptist preachers and the increase in arrests for public drunkenness during the 19th Century. Do we assume that one caused the other? No. In fact, the underlying driver was the general increase in the population — with which both were correlated.

So, remember these two things before starting your next churn analysis

If you want to know why someone churned, ask them.
If you want to analyze churn, don’t just look at who churned — compare who churned to who didn’t

5 Comments

Posted in Cloud, Metrics, SaaS

Burn Baby Burn: A Look at the Box S-1

Posted on March 27, 2014 | 14 comments

I’m pretty busy this week so I was hoping not to dive into the Box S-1, but David Cummings’ excellent summary served only to whet, as opposed to satiate, my appetite.

Perhaps it was the $168M FY14 operating loss. Maybe it was the $380M in financing raised during the last three years. Or the average quarterly burn rate of $23M. But somehow, I got sucked in.

I just had to know their CAC ratio. Of course, it’s not going to be easy to calculate. While they give us quarterly S&M expense, that’s only half the equation; we’re going to have a figure out –as best we can — quarterly new annual recurring revenue (ARR).

Billings as a Sales Metric

While many SaaS companies don’t disclose “billings,” Box does — but on an annual basis only — in their S-1.

[Click on the images to see full size.]

Billings is an attempt to triangulate on new sales (or bookings) in a SaaS company. The standard way to calculate billings is to add revenue plus change in deferred revenue.

The idea is that if you want to know how “sales” went during a given period, then revenue is not a great indicator because, in a SaaS company, revenue is an indicator of how much you sold in prior periods, not the current one. So you look at deferred revenue trying to pick up the volume of new orders. The problem is that things quickly get very complicated because (1) deferred revenue is moving both down (as past deals convert into revenue) and up (as new deals are signed) and (2) deferred revenue itself is limited only to deals that are prepaid — if a company does a constant business volume but suddenly starts doing a lot of two-year prepaids, then deferred revenue will skyrocket and if, for example, hard economic times drive loyal customers to ask for bi-annual billing, then deferred revenue will plummet, all without any “real” change in underlying subscription business. In addition, multi-year non-prepaid deals are invisible from a deferred revenue perspective (because there’s nothing, i.e., no cash prepayment, to defer).

In short, any metric built upon deferred revenue is only as a good as deferred revenue at reflecting the business.

To demonstrate the relationship between billings and new ARR, I built a model which assumes a SaaS company that starts from scratch, increases new ARR added each quarter by $500K (i.e., $500K in its first quarter, $1M in its second, $1.5M in its third), does only one-year prepaid deals, and has a 90% renewal rate. Here’s what happens.

(You can download the spreadsheet with Box financial summary and the full version of the model here. Be sure to download as an Excel file, not a PDF.)

While in year one, billings is equivalent to new ARR, as you build up the renewals base, it contributes more to revenue and muddies thing up. For a company of the above size, growth, and renewal rate, the ratio of new ARR to billings ends up 0.4.

When you take this same model and (manually) force fit the new ARR numbers to try approximate Box’s revenue and billings from 2012-2014, you get:

A CAC of ~1.6

In this case (and given my assumption set) you end up with a new-ARR/billings ratio of 0.6. To make life easier, I also calculated a new-ARR/revenue ratio (see the full sheet), which ends up around 0.8. I’ll use to this number to calculate my CAC, which comes out to between 1.5 and 1.8. While not quite an idyllic 1.o to 1.2, it’s well below 2.0 and helps explain why Box has been able to raise so much money: their growth has been deemed scalable.

Billings = Ending ARR

In reviewing my models, it’s hard not to notice that billings for a period equals ending ARR for that period. This turns out to be true under my assumption set of subscription-only (no services), one-year deals only, and everything pre-paid. Why? Because for any deal taken at any point during the year, we will recognize some percent of it (X) and the rest (Y) will go to deferred revenue. The difference between X and Y changes across the year but X+Y= the deal size at all times.

This is not true when you have consulting or do multi-year prepaid deals (which can make billings > ending ARR). It’s also not true when you do semi-annual billing (which can make billings < ending ARR).

If you assume for any given company that these factors are roughly constant, then even though uniformly inaccurate, it does provide a simple way to approximate new ARR: take the difference in ending ARR two periods, add a churn assumption, and bang you have new ARR during the period.

Key Metrics, Cashflow, and the P&L

Here are some summarized key metrics (using yellow to highlight points of interest).

Of note:

Year over year growth, while high at 97% is slowly decelerating.
Gross margins are nice at nearly 80%
Operating expenses are massive: 278% of sales in 1Q12 down to “only” 182% in 4Q14.
S&M expense are a seemingly very high 121% of revenues. This looks bad, but to really know what’s going on we need to examine the CAC, which looks pretty good.
Return on sales is -112%
That burn rate sure grabs you: $22M per quarter

In many ways you see a typical “go big or go home” cloud computing firm, burning boatloads of cash but acquiring customers in a reasonably efficient manner and doing a nice job with retention/cross-sell/up-sell as judged by their retention numbers. When you look big picture, I believe they see themselves in a winner-take-all battle vs. DropBox and in this case, the strategy — while amazingly cash consumptive — does make sense.

Here is a look at cashflow and billings:

And last, but certainly not least, here is the P&L:

Of note:

I’m always amazed by the R&D spend of seemingly simple consumer services. They spent $46M in R&D last year … on what?
The $171M in S&M expense sure grabs your attention
As does the $168M net loss!

Burn baby burn!

[Revised and expanded 3/27/14 9:18 AM]

# # #

I am not a financial analyst. I do not make buy, sell, or hold recommendations on stocks. See my FAQ for affiliations and disclaimers.

14 Comments

Posted in Cloud, Metrics, Startups, VC

The Customer Acquisition Cost (CAC) Ratio: Another Subtle SaaS Metric

Posted on December 1, 2013 | 75 comments

The software-as-a-service (SaaS) space is full of seemingly simple metrics that can quickly slip through your fingers when you try to grasp them. For example, see Measuring SaaS Renewals Rates: Way More Than Meets the Eye for a two-thousand-word post examining the many possible answers to the seemingly simple question, “what’s your renewal rate?”

In this post, I’ll do a similar examination to the slightly simpler question, “what’s your customer acquisition cost (CAC) ratio?”

I write these posts, by the way, not because I revel in the detail of calculating SaaS / cloud metrics, but rather because I cannot stand when groups of otherwise very intelligent people have long discussions based on ill-defined metrics. The first rule of metrics is to understand what they are and what they mean before entertaining long discussions and/or making important decisions about them. Otherwise you’re just counting angels on pinheads.

The intent of the CAC ratio is to determine the cost associated with acquiring a customer in a subscription business. When trying to calculate it, however, there are six key issues to consider:

Months vs. years
Customers vs. dollars
Revenue on top vs. bottom
Revenue vs. gross margin
The cost of customer success
Time periods of S&M

Months vs. Years

The first question — which relates not only to CAC but also to many other SaaS metrics: is your business inherently monthly or annual?

Since the SaaS movement started out with monthly pricing and monthly payments, many SaaS businesses conceptualized themselves as monthly and thus many of the early SaaS metrics were defined in monthly terms (e.g., monthly recurring revenue, or MRR).

While for some businesses this undoubtedly remains true, for many others – particularly in the enterprise space – the real rhythm of the business is annual. Salesforce.com, the enterprise SaaS pioneer, figured this out early on as customers actually encouraged the company to move to an annual rhythm, for among other reasons, to avoid the hassle associated with monthly billing.

Hence, many SaaS companies today view themselves as in the business of selling annual subscriptions and talk not about MRR, but ARR (annual recurring revenue).

Customers vs. Dollars

If you ask some cloud companies their CAC ratio, they will respond with a dollar figure – e.g., “it costs us $12,500 to acquire a customer.” Technically speaking, I’d call this customer acquisition cost, and not a cost ratio.

There is nothing wrong with using customer acquisition cost as a metric and, in fact, the more your business is generally consistent and the more your customers resemble each other, the more logical it is to say things like, “our average customer costs $2,400 to acquire and pays us $400/month, so we recoup our customer acquisition cost in six months.”

However, I believe that in most SaaS businesses:

The company is trying to run a “velocity” and an “enterprise” model in parallel.

The company may also be trying to run a freemium model (e.g., with a free and/or a low-price individual subscription) as well.

Ergo, your typical SaaS company might be running three business models in parallel, so wherever possible, I’d argue that you want to segment your CAC (and other metric) analysis.

In so doing, I offer a few generic cautions:

Remember to avoid the easy mistake of taking “averages of averages,” which is incorrect because it does not reflect weighting the size of the various businesses.

Remember that in a bi-modal business that the average of the two real businesses represents a fictional mathematical middle.

For example, the “weighted avg” column above is mathematically correct, but it contains relatively little information. In the same sense that you’ll never find a family with 1.8 children, you won’t find a customer with $12.7K in revenue/month. The reality is not that the company’s average months to recoup CAC is a seemingly healthy 10.8 – the reality is the company has one very nice business (SMB) where it takes only 6 months to recoup CAC and one very expensive one where it takes 30. How you address the 30-month CAC recovery is quite different from how you might try to squeeze a month or two out the 10.8.

Because customers come in so many different sizes, I dislike presenting CAC as an average cost to acquire a customer and prefer to define CAC as an average cost to acquire a dollar of annual recurring revenue.

Revenue on Top vs. Bottom

When I first encountered the CAC ratio is was in a Bessemer white paper, and it looked like this.

In English, Bessemer defined the 3Q08 CAC as the annualized amount of incremental gross margin in 3Q08 divided by total S&M expense in 2Q08 (the prior quarter).

Let’s put aside (for a while) the choice to use gross margin as opposed to revenue (e.g., ARR) in the numerator. Instead let’s focus on whether revenue makes more sense in the numerator or the denominator. Should we think of the CAC ratio as:

The amount of S&M we spend to generate $1 of revenue
The amount of revenue we get per $1 of S&M cost

To me, Bessemer defined the ratio upside down. The customer acquisition cost ratio should be the amount of S&M spent to acquire a dollar of (annual recurring) revenue.

Scale Venture Partners evidently agreed and published a metric they called the Magic Number:

Take the change in subscription revenue between two quarters, annualize it (multiply by four), and divide the result by the sales and marketing spend for the earlier of the two quarters.

This changes the Bessemer CAC to use subscription revenue, not gross margin, as well as inverts it. I think this is very close to CAC should be calculated. See below for more.

Bessemer later (kind of) conceded the inversion — while they side-stepped redefining the CAC, per se, they now emphasize a new metric called “CAC payback period” which puts S&M in the numerator.

Revenue vs. Gross Margin

While Bessemer has written some great papers on Cloud Computing (including their Top Ten Laws of Cloud Computing and Thirty Q&A that Every SaaS Revenue Leader Needs to Know) I think they have a tendency to over-think things and try to extract too much from a single metric in defining their CAC. For example, I think their choice to use gross margin, as opposed to ARR, is a mistake.

One metric should be focused on measuring one specific item. To measure the overall business, you should create a great set of metrics that work together to show the overall state of affairs.

I think of a SaaS company as a leaky bucket. The existing water level is a company’s starting ARR. During a time period the company adds water to the bucket in form of sales (new ARR), and water leaks out of the bucket in the form of churn.

If you want to know how efficient a company is at adding water to the bucket, look at the CAC ratio.
If you want to know what happens to water once in the bucket, look at the renewal rates.
If you want to know how efficiently a company runs its SaaS service, look at the subscription gross margins.

There is no need to blend the efficiency of operating the SaaS service with the efficiency of customer acquisition into a single metric. First, they are driven by different levers. Second, to do so invariably means that being good at one of them can mask being bad at the other. You are far better off, in my opinion, looking at these three important efficiencies independently.

The Cost of Customer Success

Most SaaS companies have “customer success” departments that are distinct from their customer support departments (which are accounted for in COGS). The mission of the customer success team is to maximize the renewals rate – i.e., to prevent water from leaking out of the bucket – and towards this end they typically offer a form of proactive support and adoption monitoring to ferret out problems early, fix them, and keep customers happy so they will renew their subscriptions.

In addition, the customer success team often handles basic upsell and cross-sell, selling customers additional seats or complementary products. Typically, when a sale to an existing customer crosses some size or difficultly threshold, it will be kicked back to sales. For this reason, I think of customer success as handling incidental upsell and cross-sell.

The question with respect to the CAC is what to do with the customer success team. They are “sales” to the extent that they are renewing, upselling, and cross-selling customers. However, they are primarily about ARR preservation as opposed to new ARR.

My preferred solution is to exclude both the results from and the cost of the customer success team in calculating the CAC. That is, my definition of the CAC is:

I explicitly exclude the cost customer success in the numerator and exclude the effects of churn in the denominator by looking only at the new ARR added during the quarter. This formula works on the assumption that the customer success team is selling a relatively immaterial amount of new ARR (and that their primary mission instead is ARR preservation). If that is not true, then you will need to exclude both the new ARR from customer success as well as its cost.

I like this formula because it keeps you focused on what the ratio is called: customer acquisition cost. We use revenue instead of gross margin and we exclude the cost of customer success because we are trying to build a ratio to examine one thing: how efficiently do I add new ARR to the bucket? My CAC deliberately says nothing about:

What happens to the water once S&M pours it in the bucket. A company might be tremendous at acquiring customers, but terrible at keeping them (e.g., offer a poor quality service). If you look at net change in ARR across two periods then you are including both the effects of new sales and churn. That is why I look only at new ARR.

The profitability of operating the service. A company might be great at acquiring customers but unable to operate its service at a profit. You can see that easily in subscription gross margins and don’t need to embed that in the CAC.

There is a problem, of course. For public companies you will not be able to calculate my CAC because in all likelihood customer success has been included in S&M expense but not broken out and because you can typically only determine the net change in subscription revenues and not the amounts of new ARR and churn. Hence, for public companies, the Magic Number is probably your best metric, but I’d just call it 1/CAC.

My definition is pretty close to that used by Pacific Crest in their annual survey, which uses yet another slightly different definition of the CAC: how much do you spend in S&M for a dollar of annual contract value (ACV) from a new customer?

(Note that many vendors include first-year professional services in their definition of ACV which is why I prefer ARR. Pacific Crest, however, defines ACV so it is equivalent to ARR.)

I think Pacific Crest’s definition has very much the same spirit as my own. I am, by comparison, deliberately simpler (and sloppier) in assuming that customer success not providing a lot of new ARR (which is not to say that a company is not making significant sales to its customer base – but is to say that those opportunities are handed back to the sales function.)

Let’s see the distribution of CAC ratios reported in Pacific Crest’s recent, wonderful survey:

Wow. It seems like a whole lot of math and analysis to come back and say: “the answer is 1.”

But that’s what it is. A healthy CAC ratio is around 1, which means that a company’s S&M investment in acquiring a new customer is repaid in about a year. Given COGS associated with running the service and a company’s operating expenses, this implies that the company is not making money until at least year 3. This is why higher CACs are undesirable and why SaaS businesses care so much about renewals.

Technically speaking, there is no absolute “right” answer to the CAC question in my mind. Ultimately the amount you spend on anything should be related to what it’s worth, which means we need relate customer acquisition cost to customer lifetime value (LTV).

For example, a company whose typical customer lifetime is 3 years needs to have a CAC well less than 1, whereas a company with a 10 year typical customer lifetime can probably afford a CAC of more than 2. (The NPV of a 10-year subscription increasing price at 3% with a 90% renewal rate and discount at 8% is nearly $7.)

Time Periods of S&M Expense

Let me end by taking a practical position on what could be a huge rat-hole if examined from first principles. The one part of the CAC we’ve not yet challenged is the use of the prior quarter’s sales and marketing expense. That basically assumes a 90-day sales cycle – i.e., that total S&M expense from the prior quarter is what creates ARR in the current quarter. In most enterprise SaaS companies this isn’t true. Customers may engage with a vendor over a period of a year before signing up. Rather than creating some overlapped ramp to try and better model how S&M expense turns into ARR, I generally recommend simply using the prior quarter for two reasons:

Some blind faith in offsetting errors theory. (e.g., if 10% of this quarter’s S&M won’t benefit us for a year than 10% of a year ago’s spend did the same thing, so unless we are growing very quickly this will sort of cancel out).