Category Archives: Metrics

“Always Scrubbing the Pipeline” Means “Never Scrubbing the Pipeline.”

Perhaps you’ve seen this movie:

CEO:  “Wow the quarterly pipeline dropped 20% this week.  What’s going on sales VP?”

Sales VP:  “Well, that’s because we cleaned it up this week.”

CEO:  “That sounds great, but you said that last week.”

VP of Sales: “Well, that’s because we scrubbed it then, too.”

CEO:  “So shouldn’t it have been clean after last week’s cleaning?  Why did it require so much more cleaning that it dropped another 20% this week.”

VP of Sales:  “Well, you know it’s a big job and you can’t clean up the whole pipeline in a week.”

CEO:  “Should I expect it to drop another 20% next week?”

VP of Sales:  “Uh.”

CEO:  “Soon you’re going to say that we don’t have enough to make our numbers.”

VP of Sales:  “Well, I did mean to mention that I’ve been thinking of cutting the forecast because we just don’t have enough opportunities to work on.”

CEO:  “But we started the quarter with 3.2x pipeline coverage, shouldn’t that be enough?”

VP of Sales:  “Normally, yes.  But the pipeline wasn’t really clean.  Some of those opportunities weren’t real opportunities.” [1]

CEO:  “What does ‘clean’ mean?  When does it get clean?  Once clean, how long does it stay clean.”

VP of Sales:  “Well, look our view here is that we should always be scrubbing, so we’re constantly scrubbing the pipeline, always finding new things.”

What’s wrong with this conversation?  A lot. This Sales VP:

  • Has no clear definition of a scrubbed pipeline.
  • Has no process for scrubbing the pipeline.
  • Takes no accountability for the pipeline and its quality.

In my experience, the statement “we always scrub the pipeline” means precisely one thing:  “we never scrub the pipeline.”

Should that matter?  Well, using some quick assumptions [2], the average first-line enterprise sales manager is managing pipeline that cost $50,000 to generate per rep, so if they’re managing 6-8 reps they are managing pipeline that cost the company $300,000 – $400,000.  Sales managers need to manage that pipeline.  The way to manage it is through periodic, disciplined scrubs [3].

Now some managers don’t play the “always scrubbing” card.  Instead, they say “we scrub the pipeline every week on my sales forecast call.”  But once understand what a pipeline scrub looks like and remember the purpose of a forecast call [4], you realize that it’s impossible to do both at once.

How to Properly Scrub the Pipeline

While everyone will want to take their own unique angle on how to approach this, the core of a pipeline scrub is to review all the opportunities (this quarter and out quarters) in every sales rep’s pipeline to ensure that they are classified correctly with respect to:

  • Close date (which determines what quarter pipeline it’s in)
  • Stage (along a series of well defined and verifiable stages)
  • Forecast category (e.g., forecast, commit, upside)
  • Value (following specific rules about how and when to value opportunities)

These rules should be documented in a living document called something like Pipeline Management Rules (PMR) to which managers should refer during the pipeline scrub (e.g., “Jimmy, tell me what’s the rule for picking a close date in the PMR document”).

The other important thing about pipeline scrubs is timing, because pipeline scrubs will affect your sales analytics (e.g., pipeline coverage ratios, pipeline conversion rates, stage- and forecast-category weighted expected values).  Ergo, I picked a few fixed weeks per quarter (weeks 3, 6, and 9) to present scrubbed pipeline and then we typically use the week 3 snapshot for most of our early-quarter pipeline analytics [5].

The goal of the pipeline scrub is to ensure that the entire pipeline is fairly represented with respect to those rules.  By following this disciplined procedure you can ensure that your sales forecasting and analytics are not a castle built on a sand foundation, but an edifice built on bedrock.

Notes

[1] If you haven’t gone insane yet, this one should push you over.  Wait, whose job it is to accept opportunities into the pipeline?  Sales!  Once an opportunity gets into what’s known as either “stage 2” or “sales accepted lead” status, sales doesn’t get to play that card.  This represents a total failure to accept accountability.

[2] 10 this-quarter and 10 out-quarter opportunities per rep * $2,500 mean cost per opportunity = $50,000.

[3]  I am not arguing that you can’t also clean up opportunities along the way, but that needs to be a supplement to, not a substitute for, a proper pipeline scrubbing process.

[4] A forecast call is usually focused on the current quarter and on the opportunities that are expected to close in order to make the forecast.  Thus, low-probability and out-quarter opportunities are easily overlooked.

[5] Implying of course that sales perform the scrubs during weeks 2, 5, and 8 so the resulted can be presented on Monday morning of weeks 3, 6, and 9.

The Leaky Bucket, Net New ARR, and the SaaS Growth Efficiency Index

My ears always perk up when I hear someone say “net new ARR” — because I’m trying to figure out which, of typically two, ways they are using the term:

  • To mean ARR from net new customers, in which case, I don’t know why they need the word “net” in there.  I call this new business ARR (sometimes abbreviated to newbiz ARR), and we’ll discuss this more down below.
  • To mean net change in ARR during a period, meaning for example, if you sold $2,000K of new ARR and churned $400K during a given quarter, that net new ARR would be $1,600K.  This is the correct way to use this term.

Let’s do a quick review of what I call leaky bucket analysis.  Think of a SaaS company as a leaky bucket full of ARR.

  • Every quarter, sales dumps new ARR into the bucket.
  • Every quarter, customer success does its best to keep water from leaking out.

Net new ARR is the change in the water level of the bucket.  Is it a useful metric?  Yes and no.  On the yes side:

  • Sometimes it’s all you get.  For public companies that either release (or where analysts impute) ARR, it’s all you get.  You can’t see the full leaky bucket analysis.
  • It’s useful for measuring overall growth efficiency with metrics like cash burn per dollar of net new ARR or S&M expense per dollar of net new ARR.  Recall that customer acquisition cost (CAC) focuses only on sales efficiency and won’t detect the situation where it’s cheap to add new ARR only to have it immediately leak out.

If I were to define an overall SaaS growth efficiency index (GEI), I wouldn’t do it the way Zuora does (which is effectively an extra-loaded CAC), I would define it as:

Growth efficiency index = -1 * (cashflow from operations) / (net new ARR)

In English, how much cash are you burning to generate a dollar of net new ARR.  I like this because it’s very macro.  I don’t care if you’re burning cash as a result of inefficient sales, high churn, big professional services losses, or high R&D investment.  I just want to know how much cash you’re burning to make the water level move up by one dollar.

So we can see already that net new ARR is already a useful metric, if a sometimes confused term.  However, on the no side, here’s what I don’t like about it.

  • Like any compound metric, as they say at French railroad crossings, un train peut en cacher un autre (one train can hide another).  This means that while net new ARR can highlight a problem you won’t immediately know where to go fix it — is weak net new ARR driven by a sales problem (poor new ARR), a product-driven churn problem, a customer-success-driven churn problem, or all three?

Finally, let’s end this post by taking a look and then a deeper look at the SaaS leaky bucket and how I think it’s best presented.

leaky1

For example, above, you can quickly see that a massive 167% year-over-year increase in churn ARR was the cause for weak 1Q17 net new ARR.  While this format is clear and simple, one disadvantage of this simpler format is that it hides the difference between new ARR from new customers (newbiz ARR) and new ARR from existing customers (upsell ARR).  Since that can be an important distinction (as struggling sales teams often over-rely on sales to existing customers), this slightly more complex form breaks that out as well.

leaky2

In addition to breaking out new ARR into its two sub-types, this format adds three rows of percentages, the most important of which is upsell % of new ARR, which shows to what extent your new ARR is coming from existing versus new customers.  While the “correct” value will vary as a function of your market, your business model, and your evolutionary phase, I generally believe that figures below 20% indicate that you may be failing to adequately monetize your installed base and figures above 40% indicate that you are not getting enough new business and the sales force may be too huddled around existing customers.

My SaaStr 2018 Presentation: Ten Non-Obvious Things About Scaling SaaS

Below please find the slides from the presentation I gave today at SaaStr 2018, about which I wrote a teaser blog post last week.  I hope you enjoy it as much as I enjoyed making it.

I hope to see everyone next year at SaaStr — I think it’s the preeminent software, SaaS, and startups conference.

Win Rates, Close Rates and Milestone vs. Flow Analysis

Hey, what’s your win rate?

It’s another seemingly simple question.  But, like most SaaS metrics, when you dig deeper you find it’s not.  In this post we’ll take a look at how to calculate win rates and use win rates to introduce the broader concept of milestone vs. flow analysis that applies to conversion rates across the entire sales funnel.

Let’s start with some assumptions.  Once an opportunity is accepted by sales (known as a sales-accepted opportunity, or SAL), it eventually will end up in one of three terminal states:

  • Won
  • Lost
  • Other (derailed, no decision)

Some people don’t like “other” and insist that opportunities should be exclusively either won or lost and that other is an unnecessary form of lost which should be tracked with a lost reason code as opposed to its own state.  I prefer to keep other, and call it derailed, because a competitive loss is conceptually different from a project cancellation, major delay, loss of sponsor, or a company acquisition that halts the project.  Whether you want to call it other, no decision, or derailed, I think having a third terminal state is warranted from first principles.  However, it can make things complicated.

For example, you’ll need to calculate win rates two ways:

  • Win rate, narrow = wins / (wins + losses)
  • Win rate, broad = wins / (wins + losses + derails)

Your narrow win rate tells you how good you are at beating the competition.  Your broad rates tells you how good you are at closing deals (that come to a terminal state).

Narrow win rate alone can be misleading.  If I told you a company had a 66% win rate, you might be tempted to say “time to add more salespeople and scale this thing up.”  If I told you they got the 66% win rate by derailing 94 out of every 100 opportunities it generated, won 4, and lost the other 2, then you’d say “not so fast.”  This, of course, would show up in the broad win rate of 4%.

This brings up the important question of timing.  Both these win rate calculations ignore deals that push out of a quarter.  So another degenerate case is a situation where you win 4, lose 2, derail 4, and push 90 opportunities.  In this case, narrow win rate = 66% and broad win rate = 40%.  Neither is shining a light on the problem (which, if it happens continuously, I call a rolling hairball problem.)

The issue here is thus far we’ve been performing what I call a milestone analysis.  In effect, we put observers by the side of the road at various milestones (created, won, lost, derailed) and ask them to count the number opportunities that pass by each quarter.  The issue, especially with companies that have long sales cycles, is that you have no idea of progression.  You don’t know if the opportunities that passed “win” this quarter came from the opportunities that passed “created” this quarter, or if they came from last quarter, the quarter before that, or even earlier.

Milestone analysis has two key advantages

  • It’s easy — you just need to count opportunities passing milestones
  • It’s instant — you don’t have to wait to see how things play out to generate answers

The big disadvantage is it can be misleading, because the opportunities hitting a terminal state this quarter were generated in many different time periods.  For a company with an average 9 month sales cycle, the opportunities hitting a terminal state in quarter N, were generated primarily in quarter N-3, but with some coming in quarters N-2 and N-1 and some coming in quarters N-4 and N-5.  Across that period very little was constant, for example, marketing programs and messages changed.  So a marketing effectiveness analysis would be very difficult when approached this way.

For those sorts of questions, I think it’s far better to do a cohort-based analysis, which I call a flow analysis.  Instead of looking at all the opportunities that hit a terminal state in a given time period, you go back in time, grab a cohort of opportunities (e.g., all those generated in 4Q16) and then see how they play out over time.  You go with the flow.

For marketing programs effectiveness, this is the only way to do it.  Instead of a time-based cohort, you’d take a programs-based cohort (e.g., all the opportunities generated by marketing program X), see how they play out, and then compare various programs in terms of effectiveness.

The big downside of flow analysis is you end up analyzing ancient history.  For example, if you have a 9 month average sales cycle with a wide distribution around the mean, you may need to wait 15-18 months before the vast majority of the opportunities hit a terminal state.  If you analyze too early, too many opportunities are still open.  But if you put off analysis then you may get important information, but too late.

You can compress the time window by analyzing programs effectiveness not to sales outcomes but to important steps along the funnel.  That way you could compare two programs on the basis of their ability to generate MQLs or SALs, but you still wouldn’t know whether and at what relative rate they generate actual customers.  So you could end up doubling down on a program that generates a lot of interest, but not a lot of deals.

Back to our original topic, the same concept comes up in analyzing win rates.  Regardless of which win rate you’re calculating, at most companies you’re calculating it on a milestone basis.  I find milestone-based win rates more volatile and less accurate that a flow-based SAL-to-close rate.  For example, if I were building a marketing funnel to determine how many deals I need to hit next year’s number, I’d want to use a SAL-to-close rate, not a win rate, to do so.  Why?  SAL-to-close rates:

  • Are less volatile because they’re damped by using long periods of time.
  • Are more accurate because they actually tracking what you care about — if I get 100 opportunities, how many close within a given time period.
  • Automatically factor in derails and slips (the former are ignored in the narrow win rate and the latter ignored in both the narrow and broad win rates).

Let’s look at an example.  Here’s a chart that tracks 20 opportunities, 10 generated in 1Q17 and 10 generated in 2Q17, through their entire lifetime to a terminal stage.

oppty tracking

In reality things are a lot more complicated than this picture because you have opportunities still being generated in 3Q17 through 4Q18 and you’ll have opportunities that are still in play generated in numerous quarters before 1Q17.  But to keep things simple, let’s just analyze this little slice of the world.  Let’s do a milestone-based win/loss analysis.

win-loss

First, you can see the milestone-based win/loss rates bounce around a lot.  Here it’s due in part due to law of small numbers, but I do see similar volatility in real life — in my experience win rates bounce within a fairly broad zone — so I think it’s a real issue.  Regardless of that, what’s indisputable is that in this example, this is how things will look to the milestone-based win/loss analyzer.  Not a very clear picture — and a lot to panic about in 4Q17.

Let’s look at what a flow-based cohort analysis produces.

cohort1

In this case, we analyze the cohort of opportunities generated in the year-ago quarter.  Since we only generate opportunities in two quarters, 1Q17 and 2Q17, we only have two cohorts to analyze, and we get only two sets of numbers.  The thin blue box shows in opportunity tracking chart shows the data summarized in the 1Q18 column and the thin orange box shows the data for the 2Q18 column.  Both boxes depict how 3 opportunities in each cohort are still open at the end of the analysis period (imagine you did the 1Q18 analysis in 1Q18) and haven’t come to final resolution.  The cohorts both produce a 50% narrow win rate, a 43% vs. 29% broad win rate, and a 30% vs. 20% close rate.  How good are these numbers?

Well, in our example, we have the luxury of finding the true rates by letting the six open opportunities close out over time.  By doing a flow-based analysis in 4Q18 of the 1H17 cohort, we can see that our true narrow win rate is 57%, our true broad win rate is 40%, and our close rate is also 40% (which, once everything has arrived at a terminal state, is definitionally identical to the broad win rate).

cohort7

Hopefully this post has helped you think about your funnel differently by introducing the concept of milestone- vs. flow-based analysis and by demonstrating how the same business situation results in a very different rates depending on both the choice of win rate and analysis type.

Please note that the math in this example backed me into a 40% close rate which is about double what I believe is the benchmark in enterprise software — I think 20 to 25% is a more normal range. 

 

Kellblog (Dave Kellogg) Featured on the Official SaaStr Podcast

Just a quick post to highlight the fact that last week I was the featured guest on Episode 142 of the Official SaaStr  podcast produced by the SaaStr organization run by Jason Lemkin and interviewed by a delightful young Englishman named Harry Stebbings (who also runs his own podcast entitled The Twenty Minute VC).

In the 31-minute episode — which Harry very nicely says was “probably one of his favorite interviews to record” — we cover a wide range of my favorite topics, including:

    • How I got introduced to SaaS, including my experience as an early customer of Salesforce in about 2003.
    • Challenges in scaling a software business, learned at BusinessObjects as we scaled from $30M to $1B in revenues, as well as at MarkLogic and Host Analytics.
    • My favorite SaaS metric.  If you had to pick one, I’d pick LTV/CAC.
    • Why simple churn is the best way to value the annuity of a SaaS business.
    • The loose coupling of customer satisfaction and renewal rates.
    • Why SaaS companies need to “chew gum and walk at the same time” when it comes to driving the mix of new and renewal business.
    • User-based vs. usage-based pricing in SaaS and how the latter can backfire in disincenting usage of the application.
    • My thoughts on bookings vs. ARR as a SaaS metric.  (Bookings is generally seen as a four-letter word!)
    • Why SaaS companies should make “the leaky bucket” the first four lines of their financial presentation.
    • Why I think it’s a win/win when a SaaS company gives a multi-year prepaid discount that’s less than its churn rate.
    • Why I view non-prepaid, multi-year deals as basically equivalent to renewals (just collected by finance/legal instead of customer success.)
    • Why it’s OK to “double compensate” sales and customer success on renewals and incidental upsells, and why it’s OK to pay sales on non-incidental upsells to existing customers (don’t put your farmer against someone else’s hunter).
    • Why you can’t analyze churn by analyzing churn and why you should have a rigorous taxonomy of churn.
    • My responses to Harry’s “quick fire” round questions.

You can listen to the podcast via iTunes, here.  Enjoy!

 

How to Train Your VP of Sales to Think About the Forecast

Imagine a board meeting.

Director:  What’s the forecast for new ARR this quarter?

Sales VP:  $4.3M, with a best case of $5.0M.

Director:  So what’s the most likely outcome?

Sales VP:  $4.3M.

Director:  What are you really going to do?  (The classic newb trap question.)

Sales VP:  I think we can come in North of that.

Director:  What’s the worst case?

Sales VP:  $3.5M.

Director:  What are the odds of coming in at or above the forecast? 

Sales VP:  I always make my forecast.

Director:   What do you mean by worst case?

Sales VP:  You know, well, if the stars align in a bad way – a lot of stuff would have to go wrong – but if that happened, then we could end up at $3.5M.

Director:  So, let’s say a 10% chance of being at/below the worst case?

Sales VP:  I’d say more like 5%.

Director:  What do you mean by best case?

Sales VP:  Well, if we really struck it rich and everything lined up just the way I wanted, that would be best case.

Director:  You mean if all the deals came in — so best case basically equals pipeline?

Sales VP:  No, that never happens, I’ve made about 10 scenarios of different deal closing combinations and in 2 of them I can get to the best case.

You see the problem?  Does it sound familiar?  Do you realize how much time we spend talking in board meetings about “forecast,” “best case,” and “worst case” without every discussing what we mean by those terms?

Do you see how this is compounded by the sales VP’s natural, intuitive view of the outcomes?  Do you see the obvious mathematical contradictions?  “I always make my forecast” says it’s a 100% number, but then the VP says it’s the “most likely” number which implies 50%.  Then the VP says there’s a 5% chance of coming in at/less than worst case (which is much lower) and then kind of implies that there’s a 20% chance of beating best case – but the 2 out of 10 is meaningless because it’s not a probability, it’s just a count of scenarios.  Nothing adds up.

The result is, if you’re not careful, the board ends up counting angels on pinheads.  What can we do to fix this?  It’s simple:  teach (and if need be, force) your sales VP to think probabilistically.  Ask him/her how often:

  • It is reasonable to miss the forecast.  A typical answer might be 10%.
  • It is likely to come in at/below the worst case? Typical answer, 5%.
  • It is likely to meet/beat the best case? Typical answer, 20%.

So, with those three questions, we’ve now established that we want the sales VP to give us:

  • A 90% number on being at/above the forecast
  • A 20% number on being at/above the best case
  • A 5% number on being at/below the worst case

Put differently, when the sales VP decides what number to forecast that they should be thinking:

  • I should come in under my forecast once every 2.5 years (10 quarters).
  • I should hit/beat the best case about once every 5 quarters (a bit less than once a year).
  • I should come in/under the worst case once every 20 quarters (once every 5 years, or for most minds, basically never).

The beauty here is that when you work at a company a long time you can get enough quarters under your belt, to start really seeing how you’re doing relative to these frequencies.  What’s more, by converting the probabilities into frequencies (e.g., once every 10 quarters) you make it more intuitive for the sales VP and the organization to think this way.

In addition, you have a basis for conversations like this one which, among other things, is about overconfidence:

CEO:  You need to work on your forecasting.

Sales VP:  You know it’s hard out there, very competitive, and we don’t have much deal flow.  Back when I was at { Salesforce | Oracle | SAP }, I was much better at forecasting because we had more volume.

CEO:  But we agreed your forecast should be a 90% number and you’ve missed it 2 out of the past 4 quarters.

Sales VP:  Yes, but as I’ve said it’s tough to forecast in this market.

CEO:  Then forecast a lower number so you can beat it 90% of the time.  I’m asking you for a 90% number and empirically you’re giving me a 50% number. 

Sales VP:  OK.

CEO:  Plus, when those two big deals slipped last quarter you didn’t drop your forecast, why?

Sales VP:  Because where I grew up, you don’t cut the forecast.  You try like crazy to hold it.  Do you know the morale problems it causes when I cut the forecast – especially if it’s below plan? So, yes, when those two deals slipped it added more risk to the forecast – and I told you and the board that — but I didn’t cut forecast, no. 

CEO:  But “adding risk” here is meaningless.  In reality, “adding risk” means it’s not a 90% number anymore.  You’ve taken what was a 90% number and it’s now more like a 60% or 70% number.  So I want you to forget what they taught you growing up in sales and always – every week – give me a number that based on all available information you are 90% sure you can beat.  If that means dropping the forecast so be it.

sales forecast

This also helps with the board and the inevitable sandbagger issue.  In my experience (and with a bit of exaggeration) you always seem to be in one of two situations:  (1) intermittently missing plan and in trouble or (2) consistently making plan and a “sandbagger” – it feels like there’s nothing in between.

Well, if you establish with the board that your company forecast is a 90% number it means you are supposed to beat it 9 times out of 10 so you can only really be labelled a sandbagger when you’re 15 for 15 or 20 for 20.  It also reminds them that you’re supposed to arrive at the forecast so that you miss once every 10 quarters so they shouldn’t freak out if once every 2.5 years if that happens — it’s supposed to happen in this system.  (Just don’t let a once-in-ten-quarter event happen twice in a row.)

I like this quantitative basis for sales forecasting and I carry it down to the salesrep and pipeline level.  I believe that each “forecast category” should have a probability associated with it.  For example, at the opportunity level, you should link probabilities to categories, such as:

  • Commit = 90%
  • Forecast = 70%
  • Upside = 30%

This, in turn, means that over time, a given salesrep should close 90% of their committed deals, 70% of their forecast deals, and 30% of their upside.  Deviations from this over time indicate that the rep is mis-categorizing the deals because the probability should be the basis for the forecast category assignment [1].

Finally, I do believe that salesreps should give quarterly forecasts [2] that reflect their sense for how things will come in given all the odd things that can happen to deals (e.g., size changes, acceleration, slippage).  I believe those forecasts should be a 70% number because the sales manager will be managing across a  portfolio of them and while there is little room for a company to miss at the VP of Sales level, there is more room for and more variance in performance across salesreps.

While I know this will not necessarily come naturally to all sales VPs — and some may push-back hard — this is a simple, practical, and rigorous way to think about the forecast.

# # #

[1] Some people do this through an independent (orthogonal) field in the CRM system called probability.  I think that’s unnecessary because in my mind forecast category should effectively equal probability and your options for picking a probability should be bucketed.  No one can say a deal is 43% vs. 52% and forecast category doesn’t indicate some probability of closing, then … what use is it and on what basis should you classify something as forecast vs. upside?

[2] Some people believe that only managers should make forecasts, but I believe both reps and managers should forecast for two reasons:  (1) provided it’s left independent and not “managed” by the managers, the aggregated salesrep-level forecast provides another, Wisdom of Crowds-y, view into the sales forecast and (2) it’s never too early to teach salesreps how to forecast which is best learned through the experience of trial and error over many quarters.

In-Memory Analytics: The Other Kind – A Key Success Factor for Your Career

I’m not going to talk about columnar databases, compression, horizontal partitioning, SAP Hana, or real-time vs. pre-aggregated summarization in this post on in-memory analytics.  I’m going to talk about the other kind of in-memory analytics.  The kind that can make or break your career.

What do you mean, the other kind of in-memory analytics?  Quite simply, the kind you keep in your head (i.e., in human memory).  Or, better put, the kind you should be expected to keep in your head and be able to recite on demand in any business meeting.

I remember when I worked at Salesforce, I covered for my boss a few times at the executive staff meeting when he was traveling or such.  He told me:  “Marc expects everyone to know the numbers, so before you go in there, make sure you know them.”  And I did.  On the few times I attended in his place, I made a cheat sheet and studied it for an hour to ensure that I knew every possible number that could reasonably be asked.  I’d sit in the meeting, saying little, and listening to discussion not directly related to our area.  Then, boom, out of left field, Marc asked:  “what is the Service Cloud pipeline coverage ratio for this quarter in Europe?”

“3.4,” I replied succinctly.  If I hadn’t have known the number I’m sure it would been an exercise in plucking the wings off a butterfly.  But I did, so the conversation quickly shifted to another topic, and I lived to fight another day.

Frankly, I was happy to work in an organization where executives were expected to know — in their heads, in an instant — the values of the key metrics that drive their business.  I weak organizations you constantly hear “can I get back to you on that” or “I’m going to need to look that one up.”

If you want to run a business, or a piece of one,  and you want to be a credible leader — especially in a metrics-driven organization — you need to have “in-memory” the key metrics that your higher-ups and peers would expect you to know.

This is as true of CEO pitching a venture capitalist and being asked about CAC ratios and churn rates as it is of a marketing VP being asked about keywords, costs, and conversions in an online advertising program.  Or a sales manager being asked about their forecast.

In fact, as I’ve told my sales directors a time or two:  “I should be able to wake you up at 3:00 AM and ask your forecast, upside, and pipeline and you should be able to answer, right then, instantly.”

That’s an in-memory metric.  No “let me check on that.”  No “I’ll get back to you.”  No “I don’t know, let me ask my ops guy,” which always makes me think: who runs the department, you or the ops guy — and if you need to ask the ops guy all the numbers maybe he/she should be running the department and not you?

I have bolded the word “expect” four times above because this issue is indeed about expectations and expectations are not a precise science.  So, how can you figure out the expectations for which analytics you should hold in-memory?

  • Look at your department’s strategic goals and determine which metrics best measure progress on them.
  • Ask peers inside the company what key metrics they keep in-memory and design your set by analogy.
  • Ask peers who perform the same job at different companies what key metrics they track.
  • When in doubt, ask the boss or the higher-ups what metrics they expect you to know.

Finally, I should note that I’m not a big believer in the whole “cheat sheet” approach I described above.  Because that was a special situation (covering for the boss), I think the cheat sheet was smart, but the real way to burn these metrics into your memory is to track them every week at your staff meeting, watching how they change week by week and constantly comparing them to prior periods and to a plan/model if you have one.

The point here is not “fake it until you make it” by running your business in a non-metrics-focused way and memorizing figures before a big meeting, but instead to burn the metrics review into your own weekly team meeting and then, naturally, over time you will know these metrics so instinctively that someone can wake you up at 3:00 AM and you can recite them.

That’s the other kind of in-memory analytics.  And, much as I love technology, the more important kind for your career.