Please Give Me a 10:  Interpreting Customer Satisfaction Surveys in an Era of Bias

Posted on June 10, 2015 | Leave a comment

Say you’re considering going out to dinner in a city you’ve never visited before and there are two different surveys of local restaurants that you can use to help choose a place to eat:

  • Survey 1, which is taken by randomly asking customers leaving restaurants about their experience.
  • Survey 2, which was conducted by asking every restaurant to provide 25 customers to survey.

Which would you pick?  Survey 1, every time.  Right.  It’s obvious.

Why?  Because of what they measure:

  • Survey 1 measures customer satisfaction with the restaurant in an objective way and can be used to attempt to predict your experience if you eat there. In a perfect world, you could even slice the survey results by people-like-you (e.g., who liked the same restaurants or have similar food profiles) and then it would be an even-better predictor.
  • Survey 2 measures how well the restaurant can pick, prime, and potentially bribe (e.g., “three free meals if you take the survey and give us a 10”) its top customers. It has little predictive value.  It is more a measure of how well the restaurants play the survey game than the quality of the restaurant.

Would it surprise you to know that virtually every major survey in IT software is run like Survey 2?   From big-name analyst firms to respected boutiques, the vast majority of analysts run their customer satisfaction surveys like Survey 2.

Why would they do this, when it’s so obviously invalid?  Because it’s easier, particularly when you need to include a bunch of relatively small startups.  Finding a random list of Oracle and SAP customers isn’t that hard.  Try finding 20 customers of a startup that only has 50.  You can’t do it.

So you make do and ask vendors for a list of customers to survey.  You get a lot of data you can analyze and put into reports and/or awards.  More disturbingly, you can build your special two-by-two quadrant or concentric circle diagram leveraging the data your survey, lending it more legitimacy.  (Typically these diagrams have one more-objective and one more-subjective dimension and things like revenue/size/growth and customer satisfaction factor into the more-objective dimension.)

When people challenge your survey and the methodology behind it, you can typically defend yourself in one of several ways:

  • “The data is the data; I’ve got to work with what I have.” But the data is garbage because of the biased way in which it was collected and the first rule of data is you can’t analyze garbage – “garbage in, garbage out” as the maxim goes.
  • “It was a fair contest,” meaning that every vendor had the same opportunity to select, prime, coach, cajole, reward, and/or bribe the respondents.” While this may be an excellent stiff-arm for the vendors, end users don’t care if your survey was FAIR, they care if it is VALID – i.e., can it provide a reasonable prediction of their experience.  And, back to the vendors, are such contests even fair?  A low-end,vendor with 1000 small customers can cherry-pick its customer base more easily than an enterprise vendor with 75 big ones.
  • “The results are consistent with our general experience talking to customers.” This is a weak defense because it’s both subjective and certainly confirmation biased – what analyst wants to undermine his/her own survey?  It’s also problematic because the customers who call analysts are not random.  Some serve certain verticals or departments.  Some serve big IT groups.  An echo chamber is often created in that process.

In my opinion, the single best thing these surveys do is ferret out vendors that are marketing true vaporware (e.g., a mega-vendor with a new cloud product that they’ve given free to 300 customers in order to claim market success, but since no one is actually using it, they can’t even produce the 25 references).  For that these surveys work.  For everything else?  Not so much.

The whole situation reminds of buying a car where the dealership hits you mercilessly with:

  • “Is there anything I can do to make you more satisfied today?”
  • “Is there any reason you will not give me a 10 when corporate surveys you because my compensation will fall if I get anything other than a 10 and my lovely spouse and children (in the photo on my desk) will suffer greatly if that happens?
  • “You don’t want to hurt my children do you? So please give me a 10!”

Now I can guarantee you that the net promoter score (NPS) produced by that survey will not be valid.  So why do companies do it?  Because, like it or not, it does force a conversation where the dealership asks some important, uncomfortable questions that might highlight correctable problems.

If you’re trying to force a conversation between your organization and your customers, there is probably a role for the “please give me a 10” survey.  If, however, you are genuinely trying to measure satisfaction with your products, then there is not.

So what’s a buyer to do?  If you can’t trust these surveys, then what can you trust?  I think 3 things:

  • The vendor’s wheelhouse.  While most technology vendors attempt to position as everything to everyone, despite their misguided marketing they nevertheless develop a reputation for having a wheelhouse (i.e., an area or segment which is their real strength).  These reputations get built over time and are usually accurate, so ask people “what is vendor X’s wheelhouse?”  Not what do they say they do.  Not every area in which they have one customer — but their wheelhouse.  You should see a consistent pattern over time and you can then compare your needs to the vendor’s wheelhouse.
  • Reference customers. While I believe you can cajole someone into giving you a 10 on a survey, it’s much harder to cajole someone into bogus answers on a live reference call.  The key with reference checking is to find customers like you in terms of size, complexity, problem-solved, and general requirements.
  • Your own evaluation process. If you’ve run a good evaluation process, trust it.  Don’t let some survey up-end a process where you’ve determined that product X can solve problem Y after looking at demos, possibly do some sort of proof of concept, done vendor presentations and discovery sessions, built a statement of work, etc.

Leave a comment

Posted in AR, Marketing

Bottom-Fishing Acquisitions and Catching Falling Knives

Posted on May 25, 2015 | Leave a comment

As mentioned in my recent Curse of the Megaround post, some companies that find themselves flush with cash and under heavy pressure to grow, decide to embark on dubious acquisitions to help shore up the growth story.

As one reader it put it, you can summarize your megaround post with the simple phrase “much money makes you stupid.”  And it can.  Thus, as the old saw goes, fools and their money are soon parted.

What separates good from bad acquisitions in this context?  As a general rule, I’d say that when high-growth venture-backed companies acquire firms that would otherwise best be acquired by private equity, it’s a bad thing.  Why?

Firms destined to be acquired by private equity follow a typical pattern.

  • They are old, typically 10+ years
  • They have tried multiple iterations on a strategy and none has worked
  • They have a deep stack of technology built over the years but most of which could be quickly replaced with modern, often open source, standard components
  • They tend to get strategically inverted — starting out with “what we have” as opposed to “what the market wants”
  • They have gone through several generations of management teams
  • Basically, they’re turnarounds

So private equity funds bottom-fish these opportunities, buy companies for a fraction of the total invested venture capital, scrap most of the original dream and either [1] double down on one core piece that’s working or [2] roll the company up with N adjacent companies all selling to the same buyer.

This is hard work.  This is dirty work.  This is “wet work” involving lots of headcount changes.  And private equity is good at it.   In one sense (and excluding private equity growth funds), it’s what they do.

High-flying VC backed startups are simply the wrong types of buyers to contemplate these acquisitions.  In the core business, it’s all about grow, grow, and grow.  In the acquired business, it’s all about cut, cut, cut and focus, focus, focus.  These are two very different mentalities to hold in your head at one time and the typical fail pattern is apply the grow-grow-grow mentality to the broken startup that repeatedly hasn’t-hasn’t-hasn’t.

The other failure pattern is what I call the worst-of-breed suite.  This happens when a player in space X acquires a two-bit player in space Y, hoping to “get a deal” on a cheap technology they can then sell to their customers.   The vendor is thinking “I can sell more stuff through my existing channel.”  However, the customer is thinking “I don’t want to use a worst-of-breed product just because you decided to acquire one on the cheap.”  Moreover, with easy of integration of cloud services, there is typically no real integration advantage between the cheaply acquired product and a third-party best-of-breed one.

On Wall Street, they say that bottom-fishing falling stocks is like catching falling knives.  For high-growth startups, trying to bottom-fish failed startups is pretty much the same thing.

Leave a comment

Posted in M&A, Startups, Strategy

The Curse of the Megaround

Posted on May 24, 2015 | 18 comments

With what everyone seems reluctant to call a bubble in late-stage, private financing in full swing, I thought I’d do a quick post to drill into a concept I presented in my 2015 predictions post, something I call the curse of the megaround.

We will do that by examining the forces, and the winners and losers, surrounding a megaround.  Let’s start with a hypothetical example. Company X raises $200M at $1B pre-money, giving them a $1.2B post-money valuation.

Champagne is popped, the financing is celebrated, the tech press bows, and the company is added to many unicorn trackers.

Now what happens?

  • The CEO is under immediate pressure to invest the additional capital.  If you take the rule of thumb that most venture rounds are designed to last 18-24 months, then a $200M raise implies a cash burn rate of $8 to $10M/month or $25 to $30M/quarter.  That is an enormous burn rate and in many cases it is difficult or impossible to spend that much money wisely.
  • The CEO is under heavy pressure to triple the value of the company in 2-3 years.  The investors who do these rounds are typically looking for a 3x return in 2-3 years.  So the CEO is under huge pressure to make the company worth $3.6B in 2-3 years.
  • This, in turn, means the CEO will start investing the money not only in promising growth initiatives, but also dubious ones.   Product lines are over-extended.  Geographic over-expansion occurs.  Hiring quality drops — in an attempt to not fall behind the hiring plan and lose all hope of achieving the numbers.
  • In cases, money is waste en masse in the form of dubious acquisitions, in the hope of accelerating product, employee, and customer growth.  However, the worst time to take on tricky acquisitions is when a company is already falling behind its own hypergrowth plans.
  • All of this actions were done in the name of “well, we had no hope of making the plan if we didn’t open in 12 countries, hire 200 people, add 3 product lines, and buy those 2 companies.”  So we may as well have tried as we would have been fired anyway.  At least we gave it our best shot, right?
  • This often comes to a head in a Lone Ranger moment when the board turns on the CEO.  “Didn’t we agree to that hiring plan?  Didn’t we agree to those product line extensions?  Didn’t we agree to that acquisition?” the CEO thinks.  But the board thinks differently.  “Yes, we agreed to them, but you were accountable for their success.”

Yes, being CEO can be a lonely job.  This is why I call it the curse of the megaround — because it’s certainly a curse for the CEO.  But the situation isn’t necessarily a curse for everyone.  Let’s examine the winners and losers in these situations.

Winners

  • The founders.  They get the benefit of a large investment in their company at low dilution without the downside of increased expectations and the accountability for delivering against them.
  • The private equity fund managers.  Provided the turmoil itself doesn’t kill the company and new, more realistic plans are achieved, the PE fund managers still get their 2+20 type fee structure, earning 2% a year baseline and 20% of the eventual upside as carry.  In a “more normal” world where companies went public at $300M in market cap, there would be no way to earn such heavy fees in these investments.

Losers

  • The CEO who is typically taken out back and shot along with any of the operating managers also blamed for the situation.
  • The company’s customers who are typically ignored and under-served during the years of turmoil where the company’s focus is on chasing an unreachable growth plan and not on customer service.
  • In the event the company is sold at a flat or down valuation, the common stock holders (including founders and employees) who can see their effective ownership either slashed or wiped-out by the multiple liquidation preferences often attached to the megaround.  (People love to talk about the megaround valuation, but they never seem to talk about the terms that go with it!)
  • The private equity limited partners whose returns are diminished by the very turmoil their investment created and who are stuck paying a high 2+20 fee structure with decade-ly liquidity as opposed to the 1% fee structure and daily liquidity they’d have with mutual funds if the companies were all public (as they would have been pre-Sarbox.)
  • The private equity limited partners who ultimately might well end up with a down-round as IPO.

In some situations — e.g., huge greenfield markets which can adopt a new solution quickly and easily — a megaround may well be the right answer.  But for most companies these days, I believe they are more curse than blessing.

18 Comments

Posted in Bubbles, Startups, VC

Career Advice: Simplifiers Go Far, Complexifiers Get Stuck

Posted on May 13, 2015 | 8 comments

“If you can’t explain it to a six year-old, you don’t understand it yourself.”  – Albert Einstein

There are two types of people in business:

  • Simplifiers:  who make complex things simple
  • Complexifiers:  who make simple things complex

Quick joke
Question:  What does a complexifier call a simplifier?
Answer:  “Boss.”

Somewhere, somehow, some people decided that in business you need to make everything complicated and speak using business jargon.

Well, that’s an interesting proposal and I’m not necessarily opposed to it, so let me run it up the flagpole so we can kick it around as a strawman.   Since I hear the idea has some traction in the field, let me reach out to the guys upstairs, and we’ll see if we have the bandwidth to go forward with this.  If the cost is North of $100K, I may have to backburner it, because we need to keep some dry powder pending the results of the strategy meeting — where I know we’re considering a pivot.  Right now, the long pole in the tent isn’t marketing but strategy so let’s keep lines open.  Kudos to the team for coming up with a such a great value proposition, but for now I’m afraid can’t lean in on this one.

That’s one way of hiding behind complexity: making yourself flat incomprehensible.  While that may impress your peers, your subordinates will mock you and your superiors will ask to speak to someone else.  As I argued in this post, when dealing with senior people you need to speak clearly and, above all, answer the question.  In most organizations, while jargon and doublespeak may be prevalent in middle management, they are nearly absent in the boardroom.

The other way of hiding behind complexity is not linguistic but conceptual:  finding an upstream or bigger-picture issue that will block progress at the lower level.  Consider this statement:

I’d like to cut over to the new process, but we haven’t completed the training yet.

Is this, as it appears, a valid reason for not making progress on moving to the new process or is it disguised passive resistance?  For example, I don’t want to move to the new process so I keep “having trouble” scheduling the training.  Or is it bona fide complexification?  If the training can be boiled down to a single page that everyone can read in 5 minutes, then just cut over.

Remember the old saw:

When you ask the time, some people will tell you how to build a watch.  Others will tell you how to build a Swiss Village.

My test for spotting complexifiers is look for the following pattern:

  • Slow progress on results
  • Blamed on everything being difficult or complicated
  • With a tendency to find artificial prerequisites that sound plausible, but on further examination aren’t.

Things are as complex as we want to make them. Most of the time complexity is an excuse for either not wanting or not knowing how to do something.

My advice:  strive to make things simple.  Seek to understand them.  Struggle to find apt metaphors for them.  If you’re not burning energy trying to simplify things for your audience, you are most likely a complexifier.  If so, the next time you’re about to explain to someone why something takes so long, is so complicated, or requires 5 steps to be completed, before starting, ask yourself — do I really believe this or I am making it complicated because I either don’t want or don’t know how to do it.

8 Comments

Posted in Communications, Management

Conflict Avoidance Causes Conflict: Managers Too Nice for Their own Good

Posted on April 29, 2015 | 4 comments

One thing I try to teach all new (and many old) managers is the simple, somewhat counter-intuitive rule that conflict avoidance causes conflict.

Example:  a manager has problems with an under-performing employee, but doesn’t tell them that their work is below expectations and they need to ship up.  Over time the manager gets increasingly frustrated with the work quality and eventually fires the employee abruptly in a heated conversation.

The employee has no idea his work was sub-par, had been repeatedly reassured by his manager that things were OK, had received a solid annual performance review, and thus leaves the organization angry and confused.  This results in a downstream lawsuit, with the company entering in a weak position because (due to the lack of dialog) there is little or no “paper trail” documenting the performance issues.

Now the company is looking at a legal battle that will cost or settle in the tens of thousands of dollars — if not more — all because a manager was too afraid to say “you write bad code” or “you run events poorly.”

And why did the manager never say these things?  Because they were avoiding conflict.

This pattern happens over and over in business:

  • Managers unhappy with supplier performance and simply cancelling a contract rather than trying to work together to improve service.
  • CEOs and boards getting alignment and everyone being too polite to face the issue until it comes to a major boardroom blow-up.
  • Directors quietly passed over for promotions for reasons they don’t understand until they eventually quit the company.
  • Non-native English speakers getting glass-ceiling-ed due to their communications skills, but their manager is too afraid to put the issue on the table.

In each case, conflict avoidance results in [1] a lose/lose situation and [2] more conflict.

So to phrase a slightly longer version of my rule:  conflict aversion leads to pressure build-up which leads to explosive conflict.

Why does this happen so often in Silicon Valley?

  • In general, we are all taught to be nice as we grow up (e.g., “if you haven’t got anything nice to say, then don’t say anything at all”).  Many people have trouble adapting that principle to the workplace.
  • Silicon Valley is full of introverted math and science types who enjoy working on hard conceptual problems but who, for the most part, would rather have a root canal than sit down with someone to discuss a conflictual situation.

Overcoming this isn’t easy.  Hard as it may be to believe, I used to be “nice” myself.  But then I realized that being nice wasn’t actually being nice, it was being conflict averse, avoiding tough situations to the detriment of both parties until things come to a typically explosive ending.

Most of the articles you’ll find on the web about work conflict are about peer-level conflict, such as this nice write-up on UC San Diego’s HR site.  But the most dangerous avoided conflicts are those between manager and subordinate, where the manager simply abdicates the responsibility for doing his/her job.  Over time, this will cost someone their job — either the manager for failing to manage, or the employee, often in what he/she perceives as a blind-side attack.

Here’s a nice Harvard Business Review blog post on giving negative feedback that any conflict-averse manager should read.  If your problem runs deeper, then you should read Difficult Conversations, an excellent book in its own right but also one with a title that provides part of the job description of any senior executive.

What do CEOs do for a living?  I’d argue three things:

 

4 Comments

Posted in Management, Silicon Valley