Category Archives: Modeling

Win Rates, Close Rates and Milestone vs. Flow Analysis

Hey, what’s your win rate?

It’s another seemingly simple question.  But, like most SaaS metrics, when you dig deeper you find it’s not.  In this post we’ll take a look at how to calculate win rates and use win rates to introduce the broader concept of milestone vs. flow analysis that applies to conversion rates across the entire sales funnel.

Let’s start with some assumptions.  Once an opportunity is accepted by sales (known as a sales-accepted opportunity, or SAL), it eventually will end up in one of three terminal states:

  • Won
  • Lost
  • Other (derailed, no decision)

Some people don’t like “other” and insist that opportunities should be exclusively either won or lost and that other is an unnecessary form of lost which should be tracked with a lost reason code as opposed to its own state.  I prefer to keep other, and call it derailed, because a competitive loss is conceptually different from a project cancellation, major delay, loss of sponsor, or a company acquisition that halts the project.  Whether you want to call it other, no decision, or derailed, I think having a third terminal state is warranted from first principles.  However, it can make things complicated.

For example, you’ll need to calculate win rates two ways:

  • Win rate, narrow = wins / (wins + losses)
  • Win rate, broad = wins / (wins + losses + derails)

Your narrow win rate tells you how good you are at beating the competition.  Your broad rates tells you how good you are at closing deals (that come to a terminal state).

Narrow win rate alone can be misleading.  If I told you a company had a 66% win rate, you might be tempted to say “time to add more salespeople and scale this thing up.”  If I told you they got the 66% win rate by derailing 94 out of every 100 opportunities it generated, won 4, and lost the other 2, then you’d say “not so fast.”  This, of course, would show up in the broad win rate of 4%.

This brings up the important question of timing.  Both these win rate calculations ignore deals that push out of a quarter.  So another degenerate case is a situation where you win 4, lose 2, derail 4, and push 90 opportunities.  In this case, narrow win rate = 66% and broad win rate = 40%.  Neither is shining a light on the problem (which, if it happens continuously, I call a rolling hairball problem.)

The issue here is thus far we’ve been performing what I call a milestone analysis.  In effect, we put observers by the side of the road at various milestones (created, won, lost, derailed) and ask them to count the number opportunities that pass by each quarter.  The issue, especially with companies that have long sales cycles, is that you have no idea of progression.  You don’t know if the opportunities that passed “win” this quarter came from the opportunities that passed “created” this quarter, or if they came from last quarter, the quarter before that, or even earlier.

Milestone analysis has two key advantages

  • It’s easy — you just need to count opportunities passing milestones
  • It’s instant — you don’t have to wait to see how things play out to generate answers

The big disadvantage is it can be misleading, because the opportunities hitting a terminal state this quarter were generated in many different time periods.  For a company with an average 9 month sales cycle, the opportunities hitting a terminal state in quarter N, were generated primarily in quarter N-3, but with some coming in quarters N-2 and N-1 and some coming in quarters N-4 and N-5.  Across that period very little was constant, for example, marketing programs and messages changed.  So a marketing effectiveness analysis would be very difficult when approached this way.

For those sorts of questions, I think it’s far better to do a cohort-based analysis, which I call a flow analysis.  Instead of looking at all the opportunities that hit a terminal state in a given time period, you go back in time, grab a cohort of opportunities (e.g., all those generated in 4Q16) and then see how they play out over time.  You go with the flow.

For marketing programs effectiveness, this is the only way to do it.  Instead of a time-based cohort, you’d take a programs-based cohort (e.g., all the opportunities generated by marketing program X), see how they play out, and then compare various programs in terms of effectiveness.

The big downside of flow analysis is you end up analyzing ancient history.  For example, if you have a 9 month average sales cycle with a wide distribution around the mean, you may need to wait 15-18 months before the vast majority of the opportunities hit a terminal state.  If you analyze too early, too many opportunities are still open.  But if you put off analysis then you may get important information, but too late.

You can compress the time window by analyzing programs effectiveness not to sales outcomes but to important steps along the funnel.  That way you could compare two programs on the basis of their ability to generate MQLs or SALs, but you still wouldn’t know whether and at what relative rate they generate actual customers.  So you could end up doubling down on a program that generates a lot of interest, but not a lot of deals.

Back to our original topic, the same concept comes up in analyzing win rates.  Regardless of which win rate you’re calculating, at most companies you’re calculating it on a milestone basis.  I find milestone-based win rates more volatile and less accurate that a flow-based SAL-to-close rate.  For example, if I were building a marketing funnel to determine how many deals I need to hit next year’s number, I’d want to use a SAL-to-close rate, not a win rate, to do so.  Why?  SAL-to-close rates:

  • Are less volatile because they’re damped by using long periods of time.
  • Are more accurate because they actually tracking what you care about — if I get 100 opportunities, how many close within a given time period.
  • Automatically factor in derails and slips (the former are ignored in the narrow win rate and the latter ignored in both the narrow and broad win rates).

Let’s look at an example.  Here’s a chart that tracks 20 opportunities, 10 generated in 1Q17 and 10 generated in 2Q17, through their entire lifetime to a terminal stage.

oppty tracking

In reality things are a lot more complicated than this picture because you have opportunities still being generated in 3Q17 through 4Q18 and you’ll have opportunities that are still in play generated in numerous quarters before 1Q17.  But to keep things simple, let’s just analyze this little slice of the world.  Let’s do a milestone-based win/loss analysis.

win-loss

First, you can see the milestone-based win/loss rates bounce around a lot.  Here it’s due in part due to law of small numbers, but I do see similar volatility in real life — in my experience win rates bounce within a fairly broad zone — so I think it’s a real issue.  Regardless of that, what’s indisputable is that in this example, this is how things will look to the milestone-based win/loss analyzer.  Not a very clear picture — and a lot to panic about in 4Q17.

Let’s look at what a flow-based cohort analysis produces.

cohort1

In this case, we analyze the cohort of opportunities generated in the year-ago quarter.  Since we only generate opportunities in two quarters, 1Q17 and 2Q17, we only have two cohorts to analyze, and we get only two sets of numbers.  The thin blue box shows in opportunity tracking chart shows the data summarized in the 1Q18 column and the thin orange box shows the data for the 2Q18 column.  Both boxes depict how 3 opportunities in each cohort are still open at the end of the analysis period (imagine you did the 1Q18 analysis in 1Q18) and haven’t come to final resolution.  The cohorts both produce a 50% narrow win rate, a 43% vs. 29% broad win rate, and a 30% vs. 20% close rate.  How good are these numbers?

Well, in our example, we have the luxury of finding the true rates by letting the six open opportunities close out over time.  By doing a flow-based analysis in 4Q18 of the 1H17 cohort, we can see that our true narrow win rate is 57%, our true broad win rate is 40%, and our close rate is also 40% (which, once everything has arrived at a terminal state, is definitionally identical to the broad win rate).

cohort7

Hopefully this post has helped you think about your funnel differently by introducing the concept of milestone- vs. flow-based analysis and by demonstrating how the same business situation results in a very different rates depending on both the choice of win rate and analysis type.

Please note that the math in this example backed me into a 40% close rate which is about double what I believe is the benchmark in enterprise software — I think 20 to 25% is a more normal range. 

 

The New 2017 Gartner Magic Quadrants for Cloud Strategic CPM (SCPM) and Cloud Financial CPM (FCPM) – How to Download; A Few Thoughts

For some odd reason, I always think of this scene — The New Phone Book’s Here – from an old Steve Martin comedy whenever Gartner rolls out their new Magic Quadrants (MQ) for corporate performance management (CPM). It’s probably because all of the excitement they generate.

Last year, Gartner researchers John Van Decker and Chris Iervolino kept that excitement up by making the provocative move of splitting the CPM quadrant in two — strategic CPM (SCPM) and financial CPM (FCPM). Never complacent, this year they stirred things up again by inserting the word “cloud” before the category name for each; we’ll discuss the ramifications of that in a minute.

Free Download of 2017 CPM Magic Quadrants

But first, let me provide some links where you can download the new FCPM and SCPM magic quadrants:

Significance of the New 2017 FPCM and SCPM Magic Quadrants

The biggest change this year is the insertion of the word “cloud” in the title of the magic quadrants.  This perhaps seemingly small change, like a butterfly effect, results in an entirely new world order where two of the three megavendors in the category (i.e., IBM, SAP) get displaced from market leadership due to the lack of the credibility and/or sophistication of their cloud offerings.

For example:

  • In the strategic CPM quadrant, IBM is relegated to the Visionary quadrant (bottom right) and SAP does not even make the cut.
  • In the financial CPM quadrant, IBM is relegated to the Challenger quadrant (top left) and SAP again does not even make the cut.

Well, I suppose one might then ask, well if IBM and SAP do poorly in the cloud financial and strategic CPM magic quadrants, then how do they do in the “regular” ones?

To which the answer is, there aren’t any “regular” ones; they only made cloud ones.  That’s the point.

So I view this as the mainstreaming of cloud in EPM [1].  Gartner is effectively saying a few things:

  • Who cares how much maintenance fees a vendor derives from legacy products?
  • The size of a vendor’s legacy base is independent of its position for the future.
  • The cloud is now the norm in CPM product selection, so it’s uninteresting to even produce a non-cloud MQ for CPM. The only CPM MQs are the cloud ones.

While I have plenty of beefs with Oracle as a prospective business partner — and nearly as many with their cloud EPM offerings — to their credit, they have been making an effort at cloud EPM while IBM and SAP seem to have somehow been caught off-guard, at least from an EPM perspective.

(Some of Oracle’s overall cloud revenue success is likely cloudwashing though they settled a related lawsuit with the whistleblower so we’ll never know the details.)

Unlikely Bedfellows:  Only Two Vendors are Leaders in Both FCPM and SCPM Magic Quadrants

This creates the rather odd situation where there are only two vendors in the Leaders section of both the financial and strategic CPM magic quadrants:  Host Analytics and Oracle.  That means only two vendors can provide the depth and breadth of products in the cloud to qualify for the Leaders quadrant in both the FCPM and SCPM MQ.

I know who I’d rather buy from.

In my view, Host Analytics has a more complete, mature, and proven product line – we’ve been at this a lot longer than they have — and, well, oligopolists aren’t really famous for their customer success and solutions orientation.  More infamous, in fact.  See the section of the FCPM report where it says Oracle ranks in the “bottom 25% of vendors in this MQ on ‘overall satisfaction with vendor.’”

Or how an Oracle alumni once defined “solution selling” for me:

Your problem is you are out of compliance with the license agreement and we’re going to shut down the system.  The solution is to give us money.

Nice.

For more editorial, you can read John O’Rourke’s post on the Host Analytics corporate blog.

Download the 2017 FCPM and SCPM Magic Quadrants

Or you can download the new 2017 Gartner CPM MQs here.

# # #

Notes:

[1] Gartner refers to the category as corporate performance management (CPM).  I generally refer to it as enterprise performance management (EPM), reflecting the fact that EPM software is useful not only for corporations, but other forms of organization such as not-for-profit, partnerships, government, etc.  That difference aside, I generally view EPM and CPM as synonyms.

EPM: Now More Than Ever

The theme of my presentation at past spring’s Host Analytics World was that EPM is needed in fair, foul, or uncertain weather.  While EPM is used differently in fair and foul weather scenarios, it is a critical navigational instrument to help pilot the business.

For example, in tougher times:

  • You’re constantly re-forecasting
  • You’re doing expense reduction modeling
  • You might do a zero-based budget (particularly popular among recently PE-acquired firms)
  • You’re likely to try and reduce capex (unless you see a quick rebound)
  • You’re probably making P&L, budget, and spend authority more centralized in order to keep tighter reins on the company.

In better times:

  • You model and compare new growth opportunities
  • You often build trended budgets more than bottom-up budgets
  • You adopt rolling forecasts
  • You increase capital investment and build for the future
  • You do more strategic initiatives planning
  • You decentralize P&L responsibility

These (and others) are all capabilities of a complete EPM suite.  The point is that you use that suite differently depending on the state of the business and the economy.

Well, now with the surprise election of our 45th President, Donald Trump, we can be certain of one thing:  uncertain times.

  • Will massive investments in infrastructure (including but not limited to, The Wall) happen and what effect will that have on economic growth and interest rates?
  • Will Trump deliver the promise 4% GDP growth that he’s promised or will the economy grow slower?
  • Will promised deregulation happen and if so will it accelerate economic growth?  What effects will deregulation have on key industries like financial services, energy, and raw materials?
  • What, as a result of this and foreign policies, will be the price of a barrel of oil in one year?  What effect will that have on key industries such as transportation?
  • Will Trump spark a trade war, increasing the price imports and reducing the purchasing power of low and middle-income consumers?  What effect might a trade war have on GDP growth?
  • What impact will all this have on financial markets and the cost and availability of capital?

I don’t pretend to know the answers to these questions.  I do know, however, that there is uncertainty about all of these questions– and dozens of others — that will directly impact businesses in their performance and planning.

If you cannot predict the future, you should at least be able to respond to it in agile way.

If your company takes 6 months to make a budget that gets changed once a year, you will be very exposed to surprise changes.  If you run on rolling forecasts, you will be far more agile.  If you have good EPM tools you will able to automate tasks like reporting, consolidation, and forecasting in order to free up time for the now much more important tasks of scenario planning and modeling.

Again, if you can’t know whether oil will be $40, $50, or $70 — you can at least have modeling out all three scenarios in advance so you can react quickly when it moves.

I’ve always been a big believer in planning and EPM.  And, in this uncertain environment, companies need EPM now more than ever.

The New Split CPM Magic Quadrants from Gartner

This week Gartner research vice president John Van Decker and research director Chris Iervolino took the bold move of splitting the corporate performance management (CPM), also known as enterprise performance management (EPM), magic quadrant in two.

Instead of publishing a single magic quadrant (MQ) for all of CPM, they published two MQs, one for strategic CPM and one for financial CPM, which they define as follows:

  • Strategic Corporate Performance Management (SCPM) Solutions – this includes Corporate Planning and Modeling, Integrated Financial Planning, Strategy Management, Profitability Management, and Performance Reporting.
  • Financial Corporate Performance Management (FCPM) Solutions – this includes Financial Consolidation, Financial Reporting, Management Reporting/Costing/Forecasting, Reconciliations/Close Management, Intercompany Transactions, and Disclosure Management (including XBRL tagging)

You can download these new CPM magic quadrants here.

What do I think about this?

  • It’s bold.  It’s the first time to my recollection that an MQ has included product from different categories.  Put differently, normally MQs are full of substitute products — e.g., 15 different types of butter.  Here, we have butter next to olive oil on the same MQ.
  • It’s smart.  Their uber point is that while CPM solutions are now pretty varied, that you can pretty easily classify them into more tactical/financial uses and more strategic uses.  Highlighting this by splitting the MQs does customers a service because it reminds them to think both tactically and strategically.  That’s important — and often needed in many finance departments who are struggling simply to keep up with the ongoing tactical workload.
  • It’s potentially confusing.  You can find not just substitutes but complements on the same MQ.  For example, Host Analytics and our partner Blackline are both on the FCPM MQ.  That’s cool because we both serve core finance needs.  It’s potentially confusing because we do one thing and they do another.
  • We are stoked.  Among cloud pure-play EPM vendors, Host Analytics is the only supplier listed on both MQs.   We believe this supports our contention that we have the broadest pure-play cloud EPM product line in the business.  Only Host has both!
  • In a hype-filled world, I think Gartner does a great job of seeing through the hype-haze and focusing on customers and solutions.  They do a better job than most at not being over-influenced by Halo Effects, and I suspect that’s because they spend a lot of time talking to real customers about solving real problems.

For more, see the Future of Finance blog post on the new MQs or just go ahead and download them here.

Host Analytics World 2016 EPM Keynote Address

We’re just finishing up a fantastic Host Analytics World 2016, with over 800 people gathered together in San Francisco to talk about enterprise performance management (EPM).   Here are a few pictures to give you a feel for the event.

Here’s 49ers football legend Steve Young delivering his keynote address:

IMG_3627

Here’s me delivering my keynote on EPM in fair weather and foul.

IMG_3614

Here’s an artsy shot of someone taking a picture during my keynote.

IMG_3615

And, of course, here are our mascots, Tick and Tie, stuffing bags for Project Night Night, the philanthropic activity we had at the conference cosponsored by Host Analytics and our amazing customer, Thrivent Financial.

tick and tie

The conference has been superb and I want to thank everyone — customers, prospective customers, analysts, journalists, pundits, and partners — for being a part of this great event.

I find it amazing that at such a great time to be in the cloud EPM market that we have competitors more focused on business intelligence (BI), predictive analytics, and functional performance management than on core EPM itself.  At Host Analytics, we know who we want to be:  the best vendor in cloud EPM, serving the fat middle 80% of the market.  More importantly, perhaps, we know who we don’t want to be:  we don’t want to be a visual analytics vendor, a social collaboration vendor, or a sales performance management vendor — hence our partnerships with Qlik, Socialcast, and Xactly.

We serve finance, we speak finance, and we’re proud of that.  Oh, and yes, our customers, finance leaders, care about the whole enterprise so we offer not only solutions to automate core finance processes but also tools to model the entire enterprise and align finance and operations.

You can hear about this and other topics by watching the 75 minute keynote speech and demo, embedded below.

 

Finally, please remember to save the date for Host Analytics World 2017 — May 16 through 19, 2017.

nashville

 

Video of my Host Analytics World 2015 Keynote Presentantion

Thanks to the 700+ folks who attended my keynote address at last week’s Host Analytics World 2015 conference in San Francisco.  We were thrilled with the event and thank everyone — customers, partners, and staff — who made it all possible.

Below is a 76-minute video of the keynote presentation I gave at the event. Enjoy!  And please mark your calendars now for next year’s Host Analytics World — May 9 through May 12, 2016 — in San Francisco.

Why Modeling Cloud Matters in EPM and Operational Planning

Two weeks ago, Host Analytics launched an amazing new product called Modeling Cloud.  Built by an elite team of some our most experienced developers, Modeling Cloud represents a breakthrough in cloud enterprise performance management (EPM).

In this post, I’ll discuss why Modeling Cloud matters to customers, to the market, and to the company.

Why Modeling Cloud Matters for Customers

  • Ability to build non-financial models. Planning and budgeting tools are built for planning and budgeting.  As such, you want them tied to the general ledger (GL) so, for example, you can easily get actual vs. plan for periodic reporting.  But that requires a level of financial intelligence that can become cumbersome; in a typical planning system every line needs to tie to an account in the GL, be a debit/credit account type, be associated with a legal entity, and have an associated currency.  That intelligence, which is so wonderful when making budgets, becomes baggage when you just want to build a model — for example, of bookings capacity given productivity and ramping assumptions, or new sales model given advertising spend, conversion, trial, and purchase rates. That’s why most models today are built in Excel and completely disconnected from the financial planning system.
  • Ability to integrate non-financial models. The problem with departmental Excel-based modeling is that everything ends up disconnected from the central financial planning.  Consulting can tell you what happens to billings if you hire 5 more consultants in the East and sales can tell you what happens to bookings if you hire 6 more salesreps in the East, but you need to start mailing spreadsheets around if you want to see the financial outcomes (e.g., revenue, EPS) of such changes.
  • Enterprise-wide scenario analysis. The beauty of connecting departmental modeling to the corporate financial plan is that you can perform enterprise-wide sensitivity analysis.  Say we’re thinking of making a big Eastern region push next year.  When the models all tie to the financial plan, we can see the financial outcomes for the company associated with such a push, and what it means to setting expectations with board and Wall Street.  This captures the real spirit of what is often called driver-based planning.
  • The bookings-to-revenue bridge. Models can help the finance team better forecast revenue because sales tends to be bookings-oriented whereas finance is revenue-oriented.   Everyone knows that given a pipeline of 100 opportunities there can be scores of combinations where sales hits the bookings target, but each one produces different revenue depending on the composition of the orders.   This is also, more subtly, true of sales expense because any given combination will consist of a given set of deals, for a given set of products, by a given set of saleseps, and each product may have different incentives on it, and each salesrep may be in a different stage of acceleration in their compensation plan.  By modeling bookings and doing scenario analysis of various combinations of orders, finance can better predict revenue, expense, and ultimately EPS.  In a world where a minuscule EPS miss can knock off 20% of a company’s valuation in a heartbeat, this is a critical capability.

Why Modeling Cloud Matters to the Market

  • Cloud penetration.  EPM is under penetrated by the cloud, with cloud-penetration of less than 5% today.  That means that 95% of all EPM systems sold in 2014 (between $3-4B worth) were on-premises.  By comparison, sales force automation (SFA) is about 50% cloud-penetrated.  While cloud-based planning and budgeting tools have existed for over 5 years, most cloud vendors are still working on completing their suites, with a handful introducing consolidation only in the past one to two years, and just two vendors offering a modeling engine in the cloud.  While it’s not the only factor hindering cloud penetration, rounding out cloud EPM suites will definitely help accelerate moving EPM to the cloud.
  • Market penetration.  Cloud aside, EPM is an under-penetrated market, overall. A recent survey by Grant Thornton, 40% of companies reported that they weren’t using any EPM system, relying only spreadsheets for FP&A work.  This implies the $3-4B EPM market could nearly double simply by better penetrating target customers.  And the best way to penetrate these companies is not by attacking Excel, but instead to bring an intelligent Excel strategy that makes it easy to import and build both budgets and models that are connected to the financial planning system.

grant

  • Customer penetration.  EPM is under-penetrated within EPM-consumer companies.   Many EPM customers start with a dream of true enterprise-wide planning, but fallback to EPM deployment only within finance and rely on emailed spreadsheets for the “last mile.”  That’s too bad because mailing spreadsheets is both insecure and error-prone.  This situation develops often in on-premises EPM because the hassle of deploying the software across all potential users is simply too high and because the software itself is built for finance not end users.  Cloud EPM — with cloud modeling — will help with improving customer penetration not only because it introduces new reporting and slicer/dicer options, but also because — in the case of our Modeling Cloud product — it introduces the new ability to build and manipulate sub-models which give end users the data they want — and only the data they want — without having to rely on IT for configuration.

Why Modeling Cloud Matters to Host Analytics

  • Unique position.  With Modeling Cloud in the product line, Host Analytics now has the most comprehensive EPM suite in the cloud.  If you look at our primary cloud competitors, one does low-end planning and budgeting, one does visualization and mobile, and the other does cloud modeling but has only both new and functionally thin applications for core finance.
  • The finance choice.  Host Analytics has always been the finance department’s choice when it comes to core EPM (planning, budgeting, consolidation).  That’s because experienced finance people understand the depth and breadth that we bring to the cloud and aren’t interested in buying either unproven solutions or solutions that they will outgrow.
  • The operations choice.  With Modeling Cloud, Host Analytics is now also the operations choice.  Be it sales ops, marketing ops, or services ops, Host Analytics allows ops departments to do the planning and modeling that they require — and to do so in a way that easily integrates with the core financial planning system.  This gives them the best of both worlds — the ability to build any model they could build in Excel, using Excel formulas (and even using an Excel front-end if they so desire) and to do so in a way that automatically integrates with the core financial plan.
  • The best architecture.  Only Host Analytics offers a true multi-dimensional (i.e., OLAP) backend and an architecture built atop cloud-native, dynamic, elastic, NoSQL technology where we deliver phenomenal multi-dimensional analysis and leverage modern/standard components for managing physical storage, sharding, and parallelism.  This provides us with a huge advantage going forward both in terms of productivity and scaleability.

It’s been about 2.5 years since I joined Host Analytics and I’m quite proud of the work done by our entire R&D team in industrializing the core products, introducing a new layer of solutions, and now rolling out the industry’s most innovative cloud-based modeling engine.