“All Models Are Wrong, Some Are Useful.”

“I have a map of the United States … actual size. It says, Scale: 1 mile = 1 mile. I spent last summer folding it. I also have a full-size map of the world. I hardly ever unroll it.” — Stephen Wright (comedian)

Much as we build maps as models of the physical world, we build mathematical models all the time in the business world. For example:

• A sales booking capacity model
• A marketing inverted funnel model
• A marketing attribution model

These models can be incredibly useful for planning and forecasting. They are, however, of course, wrong. They’re imperfect at prediction. They ignore important real-world factors in their desire for simplification, often relying on faith in offsetting errors. Reality rarely lands precisely where the model predicted. Which brings to mind this famous quote from the British statistician George Box.

“All models are wrong. Some are useful.” — George Box

It’s one of those quotes that, if you get it, you get it. (And then you fall in love with it.) Today, I’m hoping to bring more people into the enlightened fold by discussing Box’s quote as it pertains to three everyday go-to-market (GTM) models.

First, it’s why we don’t want models to be too precise and/or too complex. They’re not supposed to be exact. They’re not supposed to model everything, they’re supposed to be simplified. They’re just models. They’re supposed to be more useful than exact.

For example, in finance, if we need to make a precise budget that handles full GAAP accounting treatment then we do that. We map every line to a general ledger (GL) account, do GAAP treatment of revenue and expense, model depreciation and allocations, et cetera. It’s a backbreaking exercise. And when you’re done, you can’t really play with it to learn and to understand. It’s precise, but it’s unwieldy — a bit like Stephen Wright’s full-scale map of the US. It’s useful if you need to bring a full-blown budget to the board for approval, but not so useful if you’re trying to understand the interplay between sales productivity, sales ramping, and sales turnover. You’d be far better off looking at a sales bookings capacity model.

To take a different example, it’s why business school teaches you discounted cashflow (DCF) analysis for capital budgeting. DCF basically throws out GAAP and asks, what are the cashflow impacts of this project? The assumption being that if the DCFs work out, then it’s a good investment and that will eventually show up in improved GAAP results. Notably — and I was really confused by this when I first learned capital budgeting — they don’t teach you to build a 20-year detailed GAAP budget with different capital project assumptions and then do scenario analysis. Instead, they strip everything else away and ask, what are the cashflow impacts of this project versus that one?

In the rest of this post, I’ll explore Box’s quote as it relates to the three SaaS GTM models I discussed in the introduction. We’ll see that it applies quite differently to each.

Sales Bookings Capacity Models

These models calculate sales bookings based on sales hiring and staffing (including attrition), sales productivity, and sales ramping (i.e., the productivity curve new sellers follow as they spend their first few quarters at the company). Given those variables and assuming some support resources and ratios (e.g., AE/SDR), they pop out a series of quarterly bookings numbers.

While simple, these models are usually pretty precise and thus can be used for both planning and forecasting (e.g., predicting the bookings number based on actual sales bookings capacity). Thus, these are a lot useful and usually only a little wrong. In fact, some CEOs, including some big name ones I know, walk around with an even simpler version of this model in their heads: new bookings = k * (the number of sellers) where that number might be counted at the start of the year or the end of Q1. (This is what can lead to the sometimes pathological CEO belief that hiring more sellers directly leads to bookings, but hiring anything else does not, or at least only indirectly.)

Marketing Inverted Funnel Models

These models calculate the quarterly demand generation (demandgen) budget given sales booking targets, a series of conversion rates (e.g., MQL to SAL, SAL to SQL, SQL to won), and assumed phase lags between conversion points. They effectively run the sales funnel backwards, saying if we need this many deals, then we need this many SQLs, this many SALs, this many MQLs, and this many leads at various preceding time intervals.

If you’re selling anything other than toothbrushes, these models are wrong. Why? Because SaaS applications, particularly in enterprise, are high-consideration purchases that involve multiple people over sometimes prolonged periods of time. (At Salesforce, we won a massive deal on my product where the overlay rep had been chasing the deal for years, including time at his prior employer.)

These models are wrong because they treat non-linear, over-time behavior as a linear funnel. I liken the reality of the high funnel more to a popcorn machine: you’re never sure which kernel is going to pop, when, but if you add this many kernels and this much heat, then some percentage of them normally pops within N quarters. These models are a lot wrong — from first principles, by not just a little bit — but they are also a lot useful.

I think they work because of offsetting errors theory, which requires the company to be on a relatively steady growth trajectory. Sure, we’re modeling that last quarter’s MQLs are this quarter’s opportunities, and that’s not right (because many are from the quarter before that), but — as long as we’re not growing too fast or, more importantly, changing growth trajectory — that will tend to come out in the wash.

Note that if you wanted to, you could always build a more sophisticated model that took into account MQL aging — or today use an AI tool that does that for you — but you’ll still always be faced with two facts: (1) the trade-offs between model complexity and usefulness and (2) that even the more sophisticated model will still break when the growth trajectory changes or reality otherwise changes out from underneath the model. Thus, I always try to build pretty simple models and then be pretty careful in interpretation of them. Think: what’s going to break this model if it changes?

Marketing Attribution Models

I try not to write much about marketing attribution because it’s quicksand, but I’ll reluctantly dip my toe today. Before proceeding, I encourage you to take a moment to buy a Marketing Attribution is Fake News mug which is a practical, if passive-aggressive, vessel from which to drink your coffee during the next QBR or board meeting.

Marketing attribution is the attempt to assign credit for marketing-generated opportunities (itself another layer of attribution problem) to the marketing channels that generated them. In English, let’s assume we all agree that marketing generated an opportunity. But that opportunity was created at a company where 15 people over the prior 6 quarters had engaged in some marketing program in some way — e.g., clicking an ad, attending a webinar, downloading a white paper, talking to us at a conference, etc.

There are typically two levels of reduction: first, we identify one primary contact from the pool of 15 and second, we identify one marketing program that we decide gets the credit for the opportunity. Typically, people use last-touch attribution, assigning credit to the last program the primary contact engaged with before the opportunity was created. This will overcredit lower-funnel programs (e.g., executive dinners) and undercredit higher-funnel programs (e.g., clicking on an ad). Some people use first-touch attribution, reversing the problem to over-credit higher-funnel programs and under-credit lower-funnel ones. Knowing that both of those problems aren’t great, some send complexity to the rescue, using points-based attribution where each touch by each person scores one or more points, and you add up those points and then allocate credit across channels or programs on a pro rata basis. This is notionally more accurate, but the relative point assignments can be arbitrary and the veil of calculation confusion generally erodes trust in the system.

The correct way, in my humble opinion, to do attribution analysis is to approach it with humility, view it as a triangulation problem, and to make sure people absolutely understand what you’re showing them before you show it (e.g., “we’ll be looking at marketing channel performance using last-touch based attribution on the next slide and before I show it, I want to ensure that everyone understands the limits of interpretation of this approach.”) Then follow any attribution-based performance analysis with some reverse-touch analysis where you show all the touches over the prior two years, deal by deal, for a small set of deals chosen by the CRO in order to demonstrate the messy, ground-level reality of prospect interactions over time. Simply put, it’s the CMO’s job to decide how to allocate resources in this very squishy world, to make those decisions (e.g., do we do tradeshow X and do we spend $Y) in active discussion with the CRO as their partner and with a full understanding of the available data and the limitations on its interpretability. The board or the e-staff simply can’t effectively back-seat drive this process by looking at one table and saying, “OMG, tradeshow oppties cost $25K each, let’s not do any more tradeshows!” If only the optimization problem were that simple.

But, back to the Box quote. How does it apply to attribution? These models are a lot wrong, at best a little useful, and even potentially dangerous. Hence my recommendations about disclaiming the data before showing it, using triangulation to take different bearings on reality, and doing reverse-touch analysis to immediately re-ground anyone floating in a cloud of last-touch-based over-simplification.

Note that the existence of next-generation, full-funnel attribution tools such as Revsure, doesn’t radically change my viewpoint here because we are talking about the fundamental principles of models. They’re always wrong — especially when trying to model something as complex as the interactions of 20 over people at a customer with 5 people and 15 marketing programs at a company, all while those people are talking to their friends and reading blogs and seeing billboards from a vendor. I believe tools like Revsure can take the models from a lot wrong to a little wrong, and ergo improve them from potentially dangerous to useful. But you should still show the reverse-touch analysis to keep people grounded.

And Box’s quote still applies: “All models are wrong. Some are useful.” And what a lovely quote it is.