I found this story today, entitled Gartner’s Magic Quadrant Goes to Court, about ZL Technologies who, citing damages of $132M, has decided to sue Gartner over its Magic Quadrants. From ZL’s web page on the suit:
ZL Technologies, a San Jose-based IT company specializing in … enterprise software solutions for e-mail and file archiving, is challenging Gartner Group and the legitimacy of Gartner’s “Magic Quadrant.” In a complaint … ZL claims that Gartner’s use of their proprietary “Magic Quadrant” is misleading and favors large vendors with large sales and marketing budgets over smaller innovators such as ZL that have developed higher performing products.
The complaint alleges: defamation; trade libel; false advertising; unfair competition; and negligent interference with prospective economic advantage.
“Sour grapes” spring to mind as an immediate reaction. In fact, ZL concedes that they’ve been ranked in the “niche” segment of every email archiving quadrant since 2005. (Ouch!) But they nevertheless argue that bigger stakes are in play and that this is not only about ZL, but Gartner itself, technological innovation, and very nearly preservation of the American way of life. Excerpt, edited for brevity:
Regardless of how the court may decide the First Amendment arguments, ZL hopes to achieve the following …
- Fair Disclosure on Conflicts of Interest. Gartner generates its revenues from payments made by the same vendors whose products it evaluates. …
- Fair Disclosure on Evaluation Scores. The tech industry would benefit if Gartner were required to disclose more data in its evaluation process and disclose component scores so vendors know exactly where they are lacking and by how much and take corrective action. …
- Better Oversight. Gartner currently has an employee act as ombudsman to handle disagreements. The conflict of interest is self-evident in the way ZL’s concerns were summarily dismissed with little supporting evidence. ZL believes that Gartner’s immense heft and power in the marketplace necessitate careful checks and balances against abuse of power. ZL believes that if IT innovation is to remain a driver for the US economy, there must be assurances that ratings agencies such as Gartner do not subvert the competitive forces which drive innovation.
I remember a long time ago CA boycotted all Gartner research after some research-related dispute. It certain did nothing to help them: picking a fight with the movie critics always seems a risky strategy for a producer.
But it is hard to argue that Magic Quadrants are good for competition. They are inherently subjective in their assessments, they two-dimensionalize an N-dimensional problem, they encourage mental laziness on the part of customers, and –- heck –- some of us work in sectors that don’t even have a magic quadrant. What’s worse, ZL? Getting a poor ranking on an existing quadrant, or selling in a software category that Gartner doesn’t even recognize?
Since I think it’s fun to read court filings (when I have the time), let’s dig down a little deeper. The court documents are here, and I’ll embed them along the way as well. Here’s the initial complaint.
One of the many arguments made in the complaint is that Gartner doesn’t do “a single minute of independent testing of the products it purports to evaluate.” When I was younger in my career, I used to buy that argument. As I gotten older, I now realize (think: Wisdom of Crowds) that it is indeed possible to get a pretty good picture of
a product’s strengths and weaknesses simply by talking to lots of people who use it.
And that’s what Gartner does. Yes, there are no guys in lab coats doing Consumer Reports style testing. But, sometimes the guys in the lab coats measure the wrong things anyway. So while Gartner does not, to my knowledge, do hands-on testing, they neither claim to do so nor, in my estimation, is such testing strictly necessary to develop an informed opinion.
That said, a pathological case of that research model is when a vendor has very small market share. If research is done primarily through talking and there’s no one to talk to, then you’re not going to get on the map very easily.
On the other hand, I love their brass tacks description of the reality behind being labeled a “niche player”:
These MQ placements were, and are, derogatory because they are understood by technology purchasers as a warning, by Gartner, that ZL and ZL products are not good choices for enterprise email archive applications.
Yep.
Also of interest was this statement by Gartner’s ombudsman:
My sense is that there has been a relationship issue for many years with [archiving analyst] Carolyn DiCenzio and at this point it’s come down to level of trust and respect.”
I suppose there’s some logical consistency at least — if you’re going to declare war on the #1 analyst firm, well, why not make it personal as well. :-)
Let’s move on. Here’s Gartner’s response, a motion to dismiss, which is much tougher reading and more techno-legal:
Clearly, Gartner’s response is based on opinion and freedom of speech. Excerpt:
Whether plaintiff’s opinions about its product are correct, comprehensible or sincere has no legal significance; what matters is that the Complaint fails to state a claim because it attacks opinions expressed by Gartner, Inc. These opinions are constitutionally protected, in part to discourage lawsuits like this one, which are aimed at chilling the free expression of ideas and opinions.
While Gartner marketing may not love that response (imagine: “could we please defend the research as well as our right to have opinions?”) it’s not a terribly surprising one.
If nothing else, you can being to see why lawsuits cost so much money. Bear in mind the legal meters are probably running at $600/hour and they’re still debating whether the case should be immediately thrown out: it’s like dropping $20K standing on the starting line fighting about the start time for the race.
Here is ZL’s opposition to Gartner’s motion for dismissal, another 32-pages:
If you didn’t jump into the document above, let me pull out the first zinging paragraph (bolding mine):
This is a commercial case about a dominant industry player’s baseless defamation of an independent startup whose growth prospects have been crushed by the defendant’s unfair business practices. Defendant Gartner, Inc. (“Gartner”), which advises businesses on information technology decisions, exercises hegemonic control over the purchases made by a wide swath of the international corporate and governmental market. The technology Gartner says to buy is bought; what Gartner says not to buy languishes unsold, leaving its developers scrambling for the leftover market share Gartner does not dictate. The problem arises when Gartner exercises its market power recklessly, maliciously or—because of its tremendous influence—negligently. When that occurs, as it has here, innovation and competition are stifled, to the detriment of small companies who lack the resources to challenge Gartner, and to the consuming public at large.
Wow, someone turned up the rhetoric meter! At this point things are quickly getting over my legal head. There arguments seem to be largely about what is fact vs. opinion. Since I’m unable to comment on the legal issue, I’ll move on.
Finally, for the strong of legal stomach, here is Gartner’s reply to the opposition t
o the motion to dismiss. (Say that ten times fast.)
Here’s a nice summary of the counter-argument:
Try as it might, ZL cannot create a dispute where there is none. ZL alleges at great length in its Complaint (and recapitulates in its Opposition) that it has a strong product and satisfied customers. The Magic Quadrant reports do not say otherwise; the real point of contention here is not the quality of ZL’s product, but instead the subjective analytical model Gartner used to assess ZL’s market position and prospects. ZL does not contest Gartner’s basic assessments of ZL—that it has a good product but needs to expand its sales and marketing—but ZL challenges its placement on the Magic Quadrant Report because Gartner uses a “misguided analytical model” that gives “undue weight to sales and marketing.”
I have no idea how this will end. Will it be quickly thrown out of court? Will it a long drawn-out case? I don’t know. I would say that Gartner’s quadrants wield enormous power and that vendors go to great lengths to maxmize their position on them. And I’d say that you can’t judge a vendor by the quality of its technology alone. While Ingres arguably had the best database technology in the 1980s, Oracle’s sales and marketing prowess caused it to win the market and any analyst who — focused solely on the technology — would have recommended Ingres at that time would have done his customers a disservice.
I don’t know how the movie here ends, but I at least expect it to be interesting.
Interesting. You can certainly be held liable for libel if you publish false statements of fact, but this reminds me of the Google-SearchKing lawsuit (SK sued Google because Google flagged a bunch of SK's backlinks as spam which caused SK's sites to disappear from search results.) Google prevailed.Magic quadrants do seem like opinion, however if Gartner misttated facts, or made statements about a vendor with reckless disregard for the truth (especially if you can show a pattern where Gartner never does that to vendors who pay them), then maybe there's a case here.I haven't looked at a Gartner report in a while. Do their reports have a disclosure on the front stating which vendors they have a relationship with? If the FTC can make bloggers disclose "material relationships," then analyst reports should definitely have to disclose their specific financial ties too.The whole pay-to-play IT analysis industry smacks of the i-banking industry in the late 90's where banks used stock ratings as leverage to win baking deals. We know how that ended… Henry Blodget is barred from the securities industry for life.
It's probably funded via ZL's marketing budget ;-)
I say kudos to ZL for taking on Goliath. While I honor the great contributions that the analyst groups bring to the industry, I believe they have gotten stuck in the past for the things they were good at and have failed to evolve other contributions.A LOT of people place credibility on Gartner and others for decisions — and they pay handsomely for the privilege to do so. It's the latter that gets my gall. In the past Gartner's operating/research models made perfect sense for specific tests and comparisons of hardware. At one time they also made sense for large-scale software. Now, the breadth of market and scale are so diverse new models of comparison are needed.Whether or not Gartner 'claims' to do testing is less relevant than the perceived 'trust' built up over the years — the perception of sound data gathering.As to the validity of 'polling' the market — I've been on the receiving end of those 'polls'. They fall into the failure zone of what I like to differentiate from Quality and Quantity: Qualification. In many cases the people who are questioned are NOT the people with the day-to-day experience with the product. They're the people who are getting the 'blow smoke up their…' reports. Who is going to speak badly of a huge investment they begged for the financing of?Here's a reality check…why can't Gartner offer one quadrant that offers the feedback of the 'client' and another of 'consultants'? Why can't there be a more 'communal' view of the quadrant offered directly by customers, where the data points can be tied back to individuals who are willing to share their perspectives and their contextual stories on the technology?Why does anyone believe that a bunch of dots on a quadrant can tell you anything about the specific context in which the product was used?I commend ZL for simply raising the topic into view for deep debate and consideration. It's past due.
IMHO, while the Gartner MQ is far from perfect and could do with more transparency (as the Forrester Wave), it nevertheless has a lot of weight in the industry.To me shooting on the referee looks like a sign of desperation.ZL Soft should join the IIAR and download our best practice paper on how to deal with the Gartner Magic quadrants…
Ok, so fair or unfair, I am an analyst. I have worked both independently, and as a contract writer and sector researcher for most of the analyst companies. I have been offered full time analyst status at two majors and half a dozen smaller practices.So with this out of the way, and references to my portfolio at links below, I will say this about the whore stables called analyst practices, Gartner, Forrester, and the ilk: They will take money to write anything, and will place the payers in the leader's quadrant, and the non-payers in the laggard quadrant.I had several 2nd and 3rd round interviews, and offers to work at named practices – all made it clear that you write what the VP Division tell you to write, you are expected to sell via your hyperbole, and that is that – "don't make moral outrage your brand." that last was from the SVP of vendor solutions at a BIG analyst company.Now, what do these analyst companies provide that the tech press does not? What is an analyst? Here are the answers:Even the pay for play analyst whores have the focus and some measure of specialization to learn more about their sectors. That's their whole twinkle…the ability to watch, question, and analyze a particular product / industry slice.This is a good thing in enterprise technology, where the vendor claims are as useless as tits on a octopus. If the analyst is working for a non-vendor paid agency, G-d bless, but they have less resources. If they work for Gartner et al on sponsored coverage, (the quadrants), white papaers, major compilations that the vendors pay to be in…then one must comb through for the gems and the themes that are independent of the vendors particular coverage.Every analyst in the whore shops has his or her ways of dropping in editorial opinions that even run counter to the paid client's interests. They can't come out and say, "SAP SUCKS". They can, however, make allusions.The whore shops also field internally sponsored quantitative research – this is the most time and labor burdened research. Surveys and focus groups, forms campaigns, etc. Only the larger agencies can sponsor such expensive research on the front end.If your company is beat up by an analyst whore shop and they wont put you in the "Innovators" quadrant, don't be sad; hire an independent firm to write you up, take what they write, and don't refute the whore shops.If your indy analyst takes you on, use them for mare than coverage, get their strategic steering input on your sector. I know that this hard beans for an internal product manager to swallow, but if your can't get what you want from the whole analysts, or from an independent like myself, and you also eschew any sector related competitive advice…what's left for you?Or, just build the best product you can, with input from customers, and for G-d's sake, ignore the analyst magic quadrants.
Undoubtedly, Gartner is the #1 analyst in the world but am a bit skeptical about what it projects in its Magic Quadrants. I cite an example which I have come across : In the MQ for Enterprise Content Management Systems, Gartner seems to cover only COTS products always. What about products such as Alfresco , which, though open Source are excellent ECM suites. I put this argument not justifying this lawsuit but on a general plane stating that Gartner might be #1 but sometimes does leave a bit to be desired.
Hmmm. Here's the disclaimer text from a report received today. Anything about $ relationship?"The Magic Quadrant is copyrighted 2 September 2009 by Gartner, Inc. and is reused with permission. The Magic Quadrant is a graphical representation of a marketplace at and for a specific time period. It depicts Gartner's analysis of how certain vendors measure against criteria for that marketplace, as defined by Gartner. Gartner does not endorse any vendor, product or service depicted in the Magic Quadrant, and does not advise technology users to select only those vendors placed in the "Leaders" quadrant. The Magic Quadrant is intended solely as a research tool, and is not meant to be a specific guide to action. Gartner disclaims all warranties, express or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose.© 2009 Gartner, Inc. and/or its Affiliates. All Rights Reserved. Reproduction and distribution of this publication in any form without prior written permission is forbidden. The information contained herein has been obtained from sources believed to be reliable. Gartner disclaims all warranties as to the accuracy, completeness or adequacy of such information. Although Gartner's research may discuss legal issues related to the information technology business, Gartner does not provide legal advice or services and its research should not be construed or used as such. Gartner shall have no liability for errors, omissions or inadequacies in the information contained herein or for interpretations thereof. The opinions expressed herein are subject to change without notice."
The disclaimer boilerplate is is so much unkosher baloney. Let me tell what the VP of analysts says in the monthly scrum:"Talk up the book. Get your client contacts to buy in via other other sponsored projects. Let them know that we can position them to stand out in any piece of published research".
I like the idea of Gartner disclosing the nature of any fees paid by vendors in the quadrant. Everyone has a right to express an opinion. But if Gartner is taking funds from evaluated vendors it would seem reasonable that the MQ would disclose who and how much is received.
I worked on the product teams at several small companies that didn't have the resources to pay for Gartner reports let alone their consulting offerings. I even started to collect links to the publicly available Gartner reports for competitive information, etc. Later, I worked for a large company that could afford both. What a difference! At least from my experience, I saw a direct correlation between the consulting dollars spent with the Magic Quadrant position. The consulting sessions were essentially giving us the answers to the test and may have also helped to influence the survey used to develop the Magic Quadrant. IMO, the 'reports' should not only disclose who Gartner included in their research but should explicitly call out companies that have had paid consulting engagements with the analysts who are writing the reports.
I didn't know that Gartner paid so much attention to sales and marketing in their evaluation of a product's quadrant position. Large companies have large sales and marketing departments, does that automatically give them a good start regardless of the quality of the product? It would seem small companies don't have a leg to stand on. Why develop IT products if, as a small company, you already start with a significant disadvantage. It would seem to discourage innovation, as the ZL claims state. But is that a legal claim…
Nice coverage of the details on this, Dave. The case has no legal merit I can find, but as a fellow vendor, I find it amusing. Gartner's angry protestations (on one of the links you included) are a bit over the top. Without paid access to Gartner, a vendor is given 30 to 40 minutes once a year to interact with any and all analysts working on the MQ. With paid access, you can talk their ear off on an unlimited basis. It's NOT explicit "pay to play," but rather, how would they even know anything about you otherwise? It's the same with Forrester in my experience.The future of analysts and "expert opinion" is the real story here I think. I posted on that on my Contentions blog: http://bit.ly/1aTjdK
I just got an official response from Gartner: http://www.analystequity.com/?p=1293
I think you all would have enjoyed the analyst panel at SIGIR '09.But I have to agree that it's bad business to sue the referee–unless the goal is simply to generate publicity.
Thank you very much for such a detailed post. Liked the way you have put it.Iris
Considering that Gartner defines the market (there are well-over 50 quadrants) along with what is and is NOT acceptable per their standards, Gartner pretty much controls the outcome. It's up to the market to bring the right toys to Gartner's game, and pay their fees, in order to be blessed.
gartner is a firm that takes bribes in the form of "paid analysis" to write reviews. I would love to see them crushed for it or forced to admit it.as someone said early on, those who pay the most show up the best on gartner's quadrants (here's look at you, lenovo).
I'm a software engineer, and after being in the business for 25 years, any time someone tries to sell me something by referring to a Gartner report, they immediately lose all credibility with me.The suit does state what I've seen happen numerous times. Great tech has withered on the vine because 'Big Company' ranked higher even though the smaller company had better tech.
Got this today from someone at MS&L about today's hearing in the matter:"In today’s hearing on Gartner’s motion to dismiss ZL’s complaint, the court did not come to a decision. Though Gartner’s marketing collateral insists that its research is “objective, defensible and credible,” Gartner continues to assert that its research reports are effectively pure opinion, namely, opinions not based on fact. ZL believes this is clearly misleading to enterprises which expect better from Gartner than mere opinions-not-based-on-fact to resolve serious IT problems."I can't find anything online verifying it and I don't know if there are any context issues, but I thought I'd share.
Dave, very interesting writeup on the ZL-Gartner lawsuit. I guess in addition to giving Gartner money, another way to positively influence Gartner is to defend them publicly. :-)Leaving the ZL-Gartner lawsuit to the courts, the more important issue in front of the IT industry is transparency in the Gartner MQ. For example:- statement of how much each vendor is paying Gartner- statement that no analyst involved in the report or their management chain owns stock in any of the evaluated companies- evaluation criteria plus the items that did not make it into evaluation criteria and why- details on how each of the vendors scored on each of the criteria- how these scores translate into positions on the quadrant- etcOr asked another way: why would anyone not want transparency in the Gartner MQ – which as we all know affects the futures of companies and stock prices?Answer: The only people that do not want transparency are those that have something to hide…
CLEAR AND SIMPLE:GARTNER IS PAY TO PLAY.ALWAYS HAS BEEN, ALWAYS WILL BE…I HOPE THE LAWSUIT CONTINUES AND THEY ARE SHOWN TO BE THE HOOKERS THAT THEY REALLY ARE.
Alvin,I'm not sure if you're saying I'm defending the publicly to win favor or not, but just in case you missed it in the post — XML server (what my company makes) — is not a Gartner-recognized category at this time. Ergo, we don't even had a quad to be on! (Think: I cried because I had no shoes and then a person who had no feet.)Part of being innovative means you break/redefine historical category boundaries and Gartner — and all other large/segmented analyst firms — don't actually handle this so well.At Mark Logic, at Gartner we deal with 4 people: the XML person, the database person, the search person, and sometimes the ECM person. We should probably also meet with application development / appserver people but don't.So, for now, on this one, I'm happily rather neutral as I'm currently quad-less.And, what's more, I'm not sure I'd take what I wrote to be a defense; to me it was more just a perspective.Best,Dave
I have to agree w/ bizQuirk and Vern – I too have been an analyst – not for a large firm but for a smaller tier 2 firm but they are all the same, the mafia – you pay me what I want, and I'll make sure you get what you want – wink wink. I was working for a large, public company that paid Gartner for it's services. We had asked Gartner to come in to listen to one of our strategies. Carolyn was actually one of the analysts in the room. She started off by saying you need to buy this report and that report from us and said that "we" (Gartner) needs more access to your executives (implying that until all of this happens – your not really going to get much 'value' from us). In the room we all understood that to mean 'pay to play'. However, with Web 2.0 functionality becoming more predominate and with articles like this, consumers can now go to sites and 'rate' or 'vote' on their products and let the ratings, provided by end users provide the information that consumers need. In time there will be no need for Gartner.
In all fairness to Gartner (and I don't work for them)- if you are unhappy with the research, then don't buy it. If you don't feel it is valuable, then don't use it or don't listen the to the results. They are in business to make money and benefit their shareholders. I have been hearing this same argument for 10 years now- nothing has changed because there are still buyers for this research and Gartner has continued to pull away from their competition (I think they are about 3X bigger than anyone else). Obviously they are driving value somehow- otherwise why is everyone buying their research?
Anonymous is right that no one is being forced to buy research from Gartner or any other analyst. And, regardless of the quality of analysis, suing an analyst for anything shy of defamation is just dumb–which makes it no wonder that the suit was dismissed.That said, I'm surprised that the market rewards analyst firms that take money from both IT vendors and IT buyers. While I never felt my previous employer's relationship with Gartner was pay-to-play, there's no question that we had to pay for the opportunity to brief analysts–in effect, payment was necessary (though not sufficient) to get attention. Meanwhile, I'd regularly hear rumors (and see circumstantial evidence) that competitors who invested more marketing budget into analyst relations saw return on that investment.To me, it's pretty simple: analysts (and advisors in general) should decide which side they're on and only take money from that side. I'm a big fan of CMS Watch on this front–they don't take money from vendors.
People buy the research based on trust — they rely on the resources Gartner has gathered to do for them what they cannot do for themselves. But if that trust is not questionable, if Gartner is unwilling to let the results be subjected to scrutiny by others (as in 'open debate' and conversation around their findings, then I would suggest that their research has not been 'tested'.The biggest issue is that they reduce detail into a mathematical abstraction that is immediately taken out of context. Even in the same category, they compare apples against oranges — and apparently no one is the wiser of any of it (well at least not that you'd know because there is no open discussion).
I've been an analyst for Yankee Group, Robert Frances Group and Aberdeen Group and an independent during the past 30 years. I've learned that it's important to differentiate users seeking guidance via purchased research and vendors seeking to curry favor with the analysts behind that research. If users get value out of what they buy and read, more power to them. But what they buy and read is directly affected by relationships between those analysts and the vendors they cover. If those analysts are forced by their employers' business models to bias those vendor relationships in any direction, any claim to impartiality or even-handedness is and should be highly suspect. (Besides, as I've said for years, I'm not sure anyone should base their business or IT strategy on anything with the word "magic" in its title.)Meanwhile, consolidation in the analyst firm business continues, most recently with Gartner planning to buy AMR. (See my blog on this at http://bit.ly/GartnerGrabsAMR/.) At the same time, my employer, Focus (www.focus.com), innovative firms like RedMonk (www.redmonk.com) and others are trying to render traditional analyst firms irrelevant, by connecting those with expertise more directly to those who need or want it. Maybe we can eventually render the debate about "pay for play" analyst firms irrelevant — which would suit me and a growing number of analysts, users and vendors I know just fine.
It is rather interesting for me to read the post. Thanks for it. I like such topics and anything connected to this matter. BTW, why don't you change design :).
Pingback: Highlights from the 2011 Wisdom of Crowds Business Intelligence Market Study | Kellblog
Pingback: Has the magic gone from Magic Quadrants?