Category Archives: AR

How Much Should You Bet on Educating the Market?

Using the Marketing Fundamental Tension Quadrant to Map Your Demandgen and Communications Strategy

Years ago I wrote a post on what I call the fundamental tension in marketing:  the gap between what we want to say and what our audience wants to hear.

For example, let’s say we’re a supply chain software company.  Our founders are super excited about our AI/ML-based algorithms for demand prediction.  Our audience, on the other hand, barely understands AI/ML [1] and wants to hear about reducing the cost of carrying inventory and matching marketing programs to inventory levels [2].

How then should we market our supply chain software?  Let’s use the following quadrant to help.

Let’s map AI/ML as a marketing message onto this framework.  Do we care about it?  Yes, a lot.  Does our audience?  No.  We’re in Box 4:  we care and they don’t, so we conclude that must therefore educate (as we might dangerously consider them) the unwashed in order to make them care about AI/ML.  We can write a white paper entitled, The Importance of AI/ML in Supply Chain Systems.  We can run a webinar with the same title.  By the way, should we expect a lot of people to attend that webinar?  No.  Why?  Because no one cares.

Market education is hard.  That’s not to say you shouldn’t do it, but realize that you are trying, in a world of competing priorities, to add one to the list and move it up to the top.  It can be done:  digital transformation is widely viewed as business priorities today.  But that took an enormous amount of work from almost the entire software industry.  Your one startup isn’t going to change the VP of Supply Chain’s priorities overnight.

Every good demandgen leader knows it’s far easier to start with things the audience already cares about and then bridge to things your company wants to talk about.  Using the movie theatre metaphor of the prior post, you put “Reduce Inventory Costs” on the marquee and you feature “AI/ML” in a lead role in the movie.

How do you determine those priorities?  I’ll scream it:  MARKET RESEARCH.  You find existing and/or run proprietary market studies targeting your business buyers, asking about their priorities.  Then you create marketing campaigns that bridge from buyer priorities to your messages.  If you’re lucky, you’re in Box 2 and everything aligns without the bridge.  But most software marketers should spend the majority of their time in Box 1, bridging between what’s important to the audience and what’s important to the company.

If you fail to build the bridge in Box 1 you’ll have a webinar full of people of who won’t buy anything.  If you put all your investment into Box 4 you’ll run a lot of empty webinars.

The number one mistake startup marketers make is that they try educate the market on too many things.  You need to care about AI/ML.  And reporting.  And, oh by the way, analytics.  And CuteName.  And features 5, 6, and 7.  And, no, no we’re not feature-driven marketing because we remember to mention benefits somewhere.  We are evangelists.  We are storytellers!

But you’re telling stories that people don’t want to hear.

My rule is simple:  every startup should have one — and only one — Box 4 message and supporting campaigns.  Sticking with our example:

  • We should have a superb white paper on the importance of AI/ML in supply chain systems.
  • We should make claims in our PR boilerplate and About Us page related to our pioneering AI/ML in supply chain systems.
  • We should run a strong analyst relations (AR) program to get thought leaders on board with the importance of AI/ML in supply chain.
  • We should commit to this message for, by marketing standards, an extraordinarily long time; it’s literally a decade-long commitment.  So choose it wisely.

To blast through 30 years of personal industry history:  for Oracle it was row-level locking; for BusinessObjects, the semantic layer; for Endeca, the MDEX engine; for MongoDB, NoSQL [3]; for Salesforce, SaaS (branded as No Software); for Anaplan, the hypercube; for GainSight, customer success; and for Alation, the data catalog [4].

To net out the art of enterprise software marketing, it’s:

  • Stay out of Box 3
  • If you’re lucky, you’re in your Box 2 [5].  Talk about what you want to say because it’s what they want to hear.
  • Spend most of your time in Box 1, bridging from what they want to hear to what you want to say.  This keeps butts in seats at programs and primes them towards your selling agenda.
  • Make one and only one bet in Box 4, use AR to help evangelize it, and produce a small number of very high quality deliverables to tell the story.

# # #

Notes

[1] Much as I barely understand a MacPherson strut, despite having been subjected to hearing about it by years of feature-driven automotive marketing.

[2] In other words, “sell what’s on the truck.”  An old example, but likely still true:  the shirt color worn by the model in a catalog typically gets 5x the orders of any other color; so why not do color selection driven by inventory levels instead of graphic design preferences?

[3] Or, as I always preferred, MyNoSQL, simultaneously implying both cheap and easy (MySQL) and document-oriented (NoSQL).  By the way, this claim is somewhat less clear to me than the proceeding two.

[4]  The more the company is the sole pioneer of a category, the more the evangelization is about the category itself.  The more the company emerges as the leader in a competitive market, the more the evangelization is about the special sauce.  For example, I can’t even name a GainSight competitor so their message was almost purely category evangelical.  Alation, by comparison, was close to but not quite a sole pioneer so I wrestled with saying “machine-learning data catalog” (which embeds the special sauce), but settled on data catalog because they were, in my estimation, the lead category pioneer.  See my FAQ for disclaimers as I have relationships past or present with many of the companies mentioned.

[5]  Any space-pioneering application is probably in Box 2.  Any technology platform is almost always in Box 3 or 4.  Any competitive emerging space probably places you in Box 1 — i.e., needing to do a lot of bridging from more generic buyer needs to your special sauce for meeting them.

Talking Competition: Methinks Thou Doth Protest Too Much.

From time to time marketers and executives need to talk about the competition with those outside the company, including analysts, partners, and prospective investors.  In this post, we’ll cover my 4 rules for this type of communication.

Be Consistent. 
The biggest mistake people make is inconsistency, often because they’re trying to downplay a certain competitor.   Example:

“Oh, TechMo.  No, we never see them.  They’re like nowhere.  And you know their technology is really non-scaleable because it runs out of address space in the Java virtual machine.  And their list-based engine doesn’t scale because it didn’t scale when the same three founders, Mo, Larry, and Curly, did their last startup which used primarily the same idea.  And while I know they’re up to 150 employees, they must be in trouble because in the past 6 months they’ve lost their VP of Sales, Jon Smith, and their VP of Product Management, Paula Sands, and that new appexchange-like thing they launched last week, with 37 solutions, well it’s a not real either because 15 of the 37 solutions aren’t even built by partners, and they’re more prototypes than applications, and — another thing — I heard that TechMo World last week in Vegas had only 400 attendees and customers didn’t react well to the announcement they made about vertical strategy.  Yes, TechMo’s nobody to us.  We hardly ever see them.”

— Would-Be Dismissive Product Marketer.

What’s the one thing the listener is thinking on hearing all this?

“Holy Cow, these guys are tracking TechMo’s every move.  They sure know a lot about somebody they supposedly never see.”

Or, in other words, “the lady doth protest too much, methinks.”  (Hamlet.)

Don’t be this person.  Stay credible.  Be consistent.  If you’re going to be dismissive of someone, dismiss them.  But don’t try to dismiss them, then bleed fear and guilt all over the audience.  Line up your words, your attitude, and your behavior.

Cede, But Cede Carefully.
Some people say never cede anything at all, but I think that’s dangerous, particularly when dealing with sophisticated audiences like industry analysts, prospective investors, or channel partners (who work in the field every day).

I think ceding builds your credibility, but you need to be careful and precise in so doing so.  To take an old example, from BusinessObjects days:

  • Bad/sloppy:  Brio is doing pretty well.
  • Good/careful:  Brio is doing pretty well — in the USA, with companies where the end-users have a strong voice in the process, and they prioritize UI over security and administration.

It’s called positioning for a reason.  You’re supposed to be able to say what you do well, what your competitors do well, and what the difference is.  If you just go on singing “anything you can do I can do better, I can do anything better than you,” then you’re not going to build much credibility with your audience.

  • Bad/sloppy:  Competitor X seems to have some traction in the market.
  • Good/careful:  Competitor X is appearing in high-end deals, has a “fake cloud” offering, and competes well against entrenched Oracle product Y.

Don’t give competitor X an ounce more than they deserve and don’t forget to point out their limitations along the way.  When it comes to credit, give it where due, but be stingy — don’t give a drop more.

This will build your credibility in being reasonably objective.  More important, it also forces you to build some positioning.  As long you are claiming universal superiority — that no one will believe — you’re letting yourself off the hook for doing your job, in building credible positioning.

Keep Your Facts Straight
Be sure of what you say.  It’s far better to say less and be correct than to add just one more point you’re not sure of and get quickly contradicted.  Why?  Because your credibility is now in question as are all your other assertions — even the correct ones.

If you’re sure about something, then say it.  If you’re not sure but think it’s probable then weasel-word it — “we’re hearing,” “I heard from customers that,” “you can see several reviews on Glassdoor where former employees say,” or simply “we think.”  But don’t assert something as fact unless you are sure it is and you’re ready to defend it.

Read the Audience to Avoid the Blindside Hit
I warn every marketer and product manager I know about the blindside hit.  When you’re doing a briefing with hardened industry analyst on a market they’ve covered for 20 years, you’re as vulnerable to a blindside hit as an NFL quarterback.

You make some assertions, and you’re feeling good.  But you stop paying attention to the audience.  You don’t notice the body language showing that they’re not buying it anymore.  You don’t read the warning signs.  You miss the building tension in their voice.   You don’t know that the vendor you’re attacking is the analyst’s favorite and they just had a big steak dinner at the roadshow they did last week in Cleveland.

And then you make one too many false claims and then like a safety on a blitz, the analyst sees a hole in the offensive line, accelerates through it, and hits you in the back at full speed.  BOOM.  You awake a few minutes later and discover you’re strapped to a stretcher with a neck collar on and the CMO and the analyst relations director are carrying you out of the meeting.

“Sorry, Brian got a little ahead of himself, there.  Bob will take it from here.”

quarterback blindside hit

Product marketer carried out of industry analyst briefing. Don’t let this be you.

 

How To Get Your Startup a Halo

How would you like your startup to win deals not only when you win a customer evaluation, but when you tie — and even sometimes when you lose?

That sounds great.  But is it even possible?  Amazingly, yes — but you need have a halo effect working to your advantage.  What is a halo effect?  Per Wikipedia,

The halo effect is a cognitive bias in which an observer’s overall impression of a person, company, brand, or product influences the observer’s feelings and thoughts about that entity’s character or properties

There’s a great, must-read book (The Halo Effect) on the how this and eight other related effects apply in business.  The book is primarily about how the business community makes incorrect attributions about “best practices” in culture, leadership, values, and process that are subsequent to — but were not necessarily drivers of — past performance.

I know two great soundbites that summarize the phenomenon of pseudo-science in business:

  • All great companies have buildings.” Which comes from the (partly discredited) Good To Great that begins with the observation that in their study cohort of top-performing companies that all of them had buildings — and thus that simply looking for commonalities among top-performing companies was not enough; you’d have to look for distinguishing factors between top and average performers.
  • “If Marc Benioff carried a rabbit’s foot, would you?”  Which comes from a this Kellblog post where I make the point that blindly copying the habits of successful people will not replicate their outcome and, with a little help from Theodore Levitt, that while successful practitioners are intimately familiar with their own beliefs and behaviors, that they are almost definitionally ignorant of which ones helped, hindered, or were irrelevant to their own success.

Now that’s all good stuff and if you stop reading right here, you’ll hopefully avoid falling for pseudo-science in business.  That’s important.  But it misses an even bigger point.

Has your company ever won (or lost) a deal because of:

  • Perceived momentum?
  • Analyst placement on a quadrant or other market map?
  • Perceived market leadership?
  • Word of mouth as the “everyone’s using it” or “next thing” choice?
  • Perceived hotness?
  • Vibe at your events or online?
  • A certain feeling or je ne sais quoi that you were more (or less) preferred?
  • Perceived vision?

If yes, you’re seeing halo effects at work.

Halo effects are real.  Halo effects are human nature.  Halo effects are cognitive biases that tip the scales in your favor.  So the smart entrepreneur should be thinking:  how do I get one for my company?  (And the smart customer, how can I avoid being over-influenced by them?  See bottom of post.)

In Silicon Valley, a number of factors drive the creation of halo effects around a company.  Some of these are more controllable than others.  But overall, you should be thinking about how you can best combine these factors into an advantage.

  • Lineage, typically in the form of previous success at a hot company (e.g., Reid Hoffman of PayPal into LinkedIn, Dave Duffield of PeopleSoft into Workday).  The implication here (and a key part of halo effects) is that past success will lead to future success, as it sometimes does.  This one’s hard to control, but ceteris paribus, co-founding (even somewhat ex post facto) a company with an established entrepreneur will definitely help in many ways, including halo effects.
  • Investors, in one of many forms:  (1) VC’s with a strong brand name (e.g., Andreessen Horowitz), (2) specific well known venture capitalists (e.g., Doug Leone), (3) well known individual investors (e.g., Peter Thiel), and to a somewhat lesser extent (4) visible and/or famous angels (e.g., Ashton Kutcher). The implication here is obvious, that the investor’s past success is an indication of your future success.  There’s no doubt that strong investors help build halo effects indirectly through reputation; in cases they can do so directly as well via staff marketing partners designed to promote portfolio companies.
  • Investment.  In recent years, simply raising a huge amount of money has been enough to build a significant halo effect around a company, the implication being that “if they can raise that much money, then there’s got to be a pony in there somewhere.” Think Domo’s $690M or Palantir’s $2.1B.   The media loves these “go big or go home” stories and both media and customers seem to overlook the increased risk associated with staggering burn rates, the waste that having too much capital can lead to, the possibility that the investors represent “dumb money,” and the simple fact that “at scale” these businesses are supposed to be profitable.  Nevertheless, if you have the stomach, the story, and the connections to raise a dumbfounding amount of capital, it can definitely build a halo around your company.  For now, at least.
  • Valuation.  Even as the age of the unicorn starts to wane, it’s undeniable that in recent years, valuation has been a key tool to generate halos around a company.  In days of yore, valuation was a private matter, but as companies discovered they could generate hype around valuation, they started to disclose it, and thus the unicorn phenomenon was born.  As unicorn status became increasingly de rigeur, things got upside-down and companies started trading bad terms (e.g., multiple liquidation preferences, redemption rights) in order to get $1B+ (unicorn) post-money valuations.  That multiplying the price of a preferred share with superior rights by a share count that includes the number of lesser preferred and common shares is a fallacious way to arrive at a company valuation didn’t matter.  While I think valuation as a hype driver may lose some luster as many unicorns are revealed as horses in party hats (e.g., down-round IPOs), it can still be a useful tool.  Just be careful about what you trade to get it.  Don’t sell $100M worth of preferred with a ratcheted 2 moving to 3x liquidation preference — but what if someone would buy just $5M worth on those terms.  Yes, that’s a total hack, but so is the whole idea of multiplying a preferred share price times the number of common shares.  And it’s far less harmful to the company and the common stock.  Find your own middle ground / peace on this issue.
  • Growth and vision.  You’d think that industry watchers would look at a strategy and independently evaluate its merits in terms of driving future growth.  But that’s not how it works.  A key part of halo effects is misattribution of practices and performance.  So if you’ve performed poorly and have an awesome strategy, it will overlooked — and conversely.  Sadly, go-forward strategy is almost always viewed through the lens of past performance, even if that performance were driven by a different strategy or affected positively or negatively by execution issues unrelated to strategy.  A great story isn’t enough if you want to generate a vision halo effect.  You’re going to need to talk about growth numbers to prove it.  (That this leads to a pattern of private companies reporting inflated or misleading numbers is sadly no surprise.)  But don’t show up expecting to wow folks with vision. Ultimately, you’ll need to wow them with growth — which then provokes interest in vision.
  • Network.  Some companies do a nice and often quiet job of cultivating friends of the company who are thought leaders in their areas.  Many do this through inviting specific people to invest as angels.  Some do this simply through communications.  For example, one day I received an email update from Vik Singh clearly written for friends of Infer. I wasn’t sure how I got on the list, but found the company interesting and over time I got to know Vik (who is quite impressive) and ended up, well, a friend of Infer.  Some do this through advisory boards, both formal and informal.  For example, I did a little bit of advising for Tableau early on and later discovered a number of folks in my network who’d done the same thing.  The company benefitted by getting broad input on various topics and each of us felt like we were friends of Tableau.  While sort of thing doesn’t generate the same mainstream media buzz as a $1B valuation, it is a smart influencer strategy that can generate fans and buzz among the cognoscenti who, in theory at least, are opinion leaders in their chosen areas.

Before finishing the first part of this post, I need to provide a warning that halo effects are both powerful and addictive.  I seem to have a knack for competing against companies pursuing halo-driven strategies and the pattern I see typically runs like this.

  • Company starts getting some hype off good results.
  • Company starts saying increasingly aggressive things to build off the hype.
  • Analysts and press reward the hype with strong quadrant placements and great stories and blogs.
  • Company puts itself under increasing pressure to produce numbers that support the hype.

And then one of three things happens:

  1. The company continues delivering strong results and all is good, though the rhetoric and vision gets more unrelated to the business with each cycle.
  2. The company stops delivering results and is downgraded from hot-list to shit-list in the minds of the industry.
  3. The company cuts the cord with reality and starts inflating results in order to sustain the hype cycle and avoid outcome #2 above.  The vision inflates as aggressively as the numbers.

I have repeatedly had to compete against companies where claims/results were inflated to “prove” the value of bad/ordinary strategies to impress industry analysts to get strong quadrant positions to support broader claims of vision and leadership to drive more sales to inflate to even greater claimed results.  Surprisingly, I think this is usually done more in the name of ego than financial gain, but either way the story ends the same way — in terminations, lawsuits and, in one case, a jail sentence for the CEO.

Look, there are valid halo-driven strategies out there and I encourage you to try and use them to your company’s advantage — just be very careful you don’t end up addicted to halo heroin.  If you find yourself wanting to do almost anything to sustain the hype bubble, then you’ll know you’re addicted and headed for trouble.

The Customer View

Thus far, I’ve written this post entirely from the vendor viewpoint, but wanted to conclude by switching sides and offering customers some advice on how to think about halo effects in choosing vendors.   Customers should:

  • Be aware of halo effects.  The first step in dealing with any problem is understanding it exists. While supposedly technical, rational, and left-brained, technology can be as arbitrary as apparel when it comes to fashion.  If you’re evaluating vendors with halos, realize that they exist for a reason and then go understand why.  Are those drivers relevant — e.g., buying HR from Dave Duffield seems a reasonable idea.  Or are they spurious —  e.g., does it really matter that one board member invested in Facebook?  Or are they actually negative — e.g., if the company has raised $300M how crazy is their burn rate, what risk does that put on the business, and how focused will they stay on you as a customer and your problem as a market?
  •  Stay focused on your problem.  I encourage anyone buying technology to write down their business problems and high-level technology requirements before reaching out to vendors.  Hyped vendors are skilled at “changing the playing field” and trained to turn their vision into your (new) requirements.  While there certainly are cases where vendors can point out valid new requirements, you should periodically step back and do a sanity check:  are you still focused on your problem or have you been incrementally moved to a different, or greatly expanded one.  Vision is nice, but you won’t be around solve tomorrow’s problems if you can’t solve today’s.
  • Understand that industry analysts are often followers, not leaders.  If a vendor is showing you analyst support for their strategy, you need to figure out if the analyst is endorsing the strategy because of the strategy’s merits or because of the vendor’s claimed prior performance.  The latter is the definition of a halo effect and in a world full of private startups where high-quality analysts are in short supply, it’s easy to find “research” that effectively says nothing more than “this vendor is a leader because they say they’re performing really well and/or they’ve raised a lot of money.” That doesn’t tell you anything you didn’t know already and isn’t actually an independent source of information.  They are often simply amplifiers of the hype you’re already hearing.
  • Enjoy the sizzle; buy the steak.  Hype king Domo paid Alec Baldwin to make some (pretty pathetic) would-be viral videos and had Billy Beane, Flo Rida, Ludacris, and Marshawn Lynch at their user conference.  As I often say, behind any “marketing genius” is an enormous marketing budget, and that’s all you’re seeing — venture capital being directly converted into hype.  Heck, let them buy you a ticket to the show and have a great time.  Just don’t buy the software because of it — or because of the ability to invest more money in hand-grooming a handful of big-name references.  Look to meet customers like you, who have spent what you want to spend, and see if they’re happy and successful.  Don’t get handled into meeting other customers only at pre-arranged meetings.  Walk the floor and talk to regular people.  Find out how many are there for the show, or because they’re actual successful users of the software.
  • Dive into detail on the proposed solution.  Hyped vendors will often try to gloss over solutions and sell you the hype (e.g., “of course we can solve your problem, we’ve got the most logos, Gartner says we’re the leader, there’s an app for that.”)  What you need is a vendor who will listen to your problem, discuss it with you intelligently, and provide realistic estimates on what it takes to solve it.  The more willing they are to do that, the better off you are.  The more they keep talking about the founder’s escape from communism, the pedigree of their investors, their recent press coverage, or the amount of capital they’ve raised, the more likely you are to end up high and dry.  People interested in solving your problem will want to talk about your problem.
  • Beware the second-worst outcome:  the backwater.  Because hyped vendors are actually serving Sand Hill Road and/or Wall Street more than their customers, they pitch broad visions and huge markets in order to sustain the halo.  For a customer, that can be disastrous because the vendor may view the customer’s problems as simply another lily pad to jump off on the path to success.  The second-worst outcome is when you buy a solution and then vendor takes your money and invests it in solving other problems.  As a customer, you don’t want to marry your vendor’s fling.  You want to marry their core.  For startups, the pattern is typically over-expansion into too many things, getting in trouble, and then retracting hard back into the core, abandoning customers of the new, broader initiatives.  The second-worst outcome is when you get this alignment wrong and end up in a backwater or formerly-strategic area of your supplier’s strategy.
  • Avoid the worst outcome:  no there there.  Once in awhile, there is no “there there” behind some very hyped companies despite great individual investors, great VCs, strategic alliances, and a previously experienced team.  Perhaps the technology vision doesn’t pan out, or the company switches strategies (“pivots”) too often.  Perhaps the company just got too focused on its hype and not on it customers.  But the worst outcome, while somewhat rare, is when a company doesn’t solve its advertised problem. They may have a great story, a sexy demo, and some smart people — but what they lack is a core of satisfied customers solving the problem the company talks about.  In EPM, with due respect and in my humble opinion, Tidemark fell into this category, prior to what it called a “growth investment” and what sure seemed to me like a (fire) sale, to Marlin Equity Partners.  Customers need to watch out for these no-there-there situations and the best way to do that is taking strong dose of caveat emptor with a nose for “if it sounds too good to be true, then it might well possibly be.”

Please Give Me a 10:  Interpreting Customer Satisfaction Surveys in an Era of Bias

Say you’re considering going out to dinner in a city you’ve never visited before and there are two different surveys of local restaurants that you can use to help choose a place to eat:

  • Survey 1, which is taken by randomly asking customers leaving restaurants about their experience.
  • Survey 2, which was conducted by asking every restaurant to provide 25 customers to survey.

Which would you pick?  Survey 1, every time.  Right.  It’s obvious.

Why?  Because of what they measure:

  • Survey 1 measures customer satisfaction with the restaurant in an objective way and can be used to attempt to predict your experience if you eat there. In a perfect world, you could even slice the survey results by people-like-you (e.g., who liked the same restaurants or have similar food profiles) and then it would be an even-better predictor.
  • Survey 2 measures how well the restaurant can pick, prime, and potentially bribe (e.g., “three free meals if you take the survey and give us a 10”) its top customers. It has little predictive value.  It is more a measure of how well the restaurants play the survey game than the quality of the restaurant.

Would it surprise you to know that virtually every major survey in IT software is run like Survey 2?   From big-name analyst firms to respected boutiques, the vast majority of analysts run their customer satisfaction surveys like Survey 2.

Why would they do this, when it’s so obviously invalid?  Because it’s easier, particularly when you need to include a bunch of relatively small startups.  Finding a random list of Oracle and SAP customers isn’t that hard.  Try finding 20 customers of a startup that only has 50.  You can’t do it.

So you make do and ask vendors for a list of customers to survey.  You get a lot of data you can analyze and put into reports and/or awards.  More disturbingly, you can build your special two-by-two quadrant or concentric circle diagram leveraging the data your survey, lending it more legitimacy.  (Typically these diagrams have one more-objective and one more-subjective dimension and things like revenue/size/growth and customer satisfaction factor into the more-objective dimension.)

When people challenge your survey and the methodology behind it, you can typically defend yourself in one of several ways:

  • “The data is the data; I’ve got to work with what I have.” But the data is garbage because of the biased way in which it was collected and the first rule of data is you can’t analyze garbage – “garbage in, garbage out” as the maxim goes.
  • “It was a fair contest,” meaning that every vendor had the same opportunity to select, prime, coach, cajole, reward, and/or bribe the respondents.” While this may be an excellent stiff-arm for the vendors, end users don’t care if your survey was FAIR, they care if it is VALID – i.e., can it provide a reasonable prediction of their experience.  And, back to the vendors, are such contests even fair?  A low-end,vendor with 1000 small customers can cherry-pick its customer base more easily than an enterprise vendor with 75 big ones.
  • “The results are consistent with our general experience talking to customers.” This is a weak defense because it’s both subjective and certainly confirmation biased – what analyst wants to undermine his/her own survey?  It’s also problematic because the customers who call analysts are not random.  Some serve certain verticals or departments.  Some serve big IT groups.  An echo chamber is often created in that process.

In my opinion, the single best thing these surveys do is ferret out vendors that are marketing true vaporware (e.g., a mega-vendor with a new cloud product that they’ve given free to 300 customers in order to claim market success, but since no one is actually using it, they can’t even produce the 25 references).  For that these surveys work.  For everything else?  Not so much.

The whole situation reminds of buying a car where the dealership hits you mercilessly with:

  • “Is there anything I can do to make you more satisfied today?”
  • “Is there any reason you will not give me a 10 when corporate surveys you because my compensation will fall if I get anything other than a 10 and my lovely spouse and children (in the photo on my desk) will suffer greatly if that happens?
  • “You don’t want to hurt my children do you? So please give me a 10!”

Now I can guarantee you that the net promoter score (NPS) produced by that survey will not be valid.  So why do companies do it?  Because, like it or not, it does force a conversation where the dealership asks some important, uncomfortable questions that might highlight correctable problems.

If you’re trying to force a conversation between your organization and your customers, there is probably a role for the “please give me a 10” survey.  If, however, you are genuinely trying to measure satisfaction with your products, then there is not.

So what’s a buyer to do?  If you can’t trust these surveys, then what can you trust?  I think 3 things:

  • The vendor’s wheelhouse.  While most technology vendors attempt to position as everything to everyone, despite their misguided marketing they nevertheless develop a reputation for having a wheelhouse (i.e., an area or segment which is their real strength).  These reputations get built over time and are usually accurate, so ask people “what is vendor X’s wheelhouse?”  Not what do they say they do.  Not every area in which they have one customer — but their wheelhouse.  You should see a consistent pattern over time and you can then compare your needs to the vendor’s wheelhouse.
  • Reference customers. While I believe you can cajole someone into giving you a 10 on a survey, it’s much harder to cajole someone into bogus answers on a live reference call.  The key with reference checking is to find customers like you in terms of size, complexity, problem-solved, and general requirements.
  • Your own evaluation process. If you’ve run a good evaluation process, trust it.  Don’t let some survey up-end a process where you’ve determined that product X can solve problem Y after looking at demos, possibly do some sort of proof of concept, done vendor presentations and discovery sessions, built a statement of work, etc.