The Pillorying of MarkLogic: Why Selling Disruptive Technology To the Government is Hard and Risky

There’s a well established school of thought that high-tech startups should focus on a few vertical markets early in their development.  The question is whether government should be one of them?

The government seems to think so.  They run a handful of programs to encourage startups to focus on government.  Heck, the CIA even has a venture arm right on Sand Hill Road, In-Q-Tel, whose mission is to find startups who are not focused on the Intelligence Community (IC) and to help them find initial customers (and provide them with a dash of venture capital) to encourage them to do so.

When I ran MarkLogic between mid-2004 and 2010, we made the strategic decision to focus on government as one of our two key verticals.  While it was then, and still is, rather contrarian to do so, we nevertheless decided to focus on government for several reasons.

  • The technology fit was very strong.  There are many places in government, including the IC, where they have a bona fide need for a hybrid database / search engine, such as MarkLogic.
  • Many people in government were tired of the Oracle-led oligopoly in the RDBMS market and were seeking alternatives.  (Think:  I’m tired of writing Oracle $40M checks.)  While this was true in other markets, it was particularly true in government because their problems were compounded by lack of good technical fit — i.e., they were paying an oligopolist a premium price for technology that was not, in the end, terribly well suited to what they were doing.
  • Unlike other markets (e.g., Finance, Web 2.0) where companies could afford the high-caliber talent able to use the then-new open source NoSQL alternatives, government — with the exception of the IC — was not swimming in such talent.  Ergo, government really needed a well-supported enterprise NoSQL system usable by a more typical engineer.

The choice had always made me nervous for a number of reasons:

  • Government deals were big, so it could lead to feast-or-famine revenue performance unless you were able to figure out how to smooth out the inherent volatility.
  • Government deals ran through systems integrators (SI) which could greatly complexify the sales cycle.
  • Government was its own tribe, with its own language, and its own idiosyncrasies (e.g., security clearances).  While bad from the perspective of commercial expansion, these things also served as entry barriers that, once conquered, should provide a competitive advantage.

The only thing I hadn’t really anticipated was the politics.

It had never occurred to me, for example, that in a $630M project — where MarkLogic might get maybe $5 to $10M — that someone would try to blame failure of what appears to be one of the worst-managed projects in recent history on a component that’s getting say 1% of the fees.

It makes no sense.  But now, for the second time, the New York Times has written an article about the fiasco where MarkLogic is not only one of very few vendors even mentioned but somehow implicated in the failures because it is different.

Let me start with a few of my own observations on from the sidelines.  (Note that I, to my knowledge, was never involved with the project during my time at MarkLogic.)

From the cheap seats the problems seem simple:

  • Unattainable timelines.  You don’t build a site “just like” using government contractors in a matter of quarters.  Amazon has been built over the course of a more than a decade.
  • No Beta program.  It’s incomprehensible to me that such a site would go directly from testing into production without quarters of Beta.  (Remember, not so long ago, that Google ran Beta’s for years?)
  • No general oversight.  It seems that there was no one playing the general contractor role.  Imagine if you built a house with plumbers, carpenters, and electricians not coordinated by a strong central resource.
  • Insufficient testing.  The absent Beta program aside, it seems the testing phase lasted only weeks, that certain basic functionality was not tested, and that it’s not even clear if there was a code-freeze before testing.
  • Late changes.  Supporting the idea that there was no code freeze are claims that the functional spec was changing weeks before the launch.

Sadly, these are not rare problems on a project of this scale.  This kind of stuff happens all the time, and each of these problems is a hallmark of a “train wreck” software development project.

To me, guessing from a distance, it seems pretty obvious what happened.

  • Someone who didn’t understand how hard it to build was ordered up a website of very high complexity with totally unrealistic timeframes.
  • A bunch of integrators (and vendors) who wanted their share of the $630M put in bids, probably convincing themselves in each part of the system that if things went very well that they could maybe make the deadlines or, if not, maybe cut some scope.  (Remember you don’t win a $50M bid by saying “the project is crazy and the timeframe unrealistic.”)
  • Everybody probably did their best but knew deep down that the project was failing.
  • Everyone was afraid to admit that the project was failing because nobody likes to deliver bad news, and it seems that there was no one central coordinator whose job it was to do so.

Poof.  It happens all the time.  It’s why the world has generally moved away from big-bang projects and towards agile methodologies.

While sad, this kind of story happens.  The question is how does the New York Times end up writing two articles where somehow the failure is somehow blamed on MarkLogic.  Why is MarkLogic even mentioned?  This the story of a project run amok, not the story of a technology component failure.

Politics and Technology

The trick with selling disruptive technology to the government is that you encounter two types of people.

  • Those who look objectively at requirements and try to figure out which technology can best do the job.  Happily, our government contains many of these types of people.
  • Those who look at their own skill sets and view any disruptive technology as a threat.

I met many Oracle-DBA-lifers during my time working with the government.  And I’m OK with their personal decision to stop learning, not refresh their skills, not stay current on technology, and to want to ride a deep expertise in the Oracle DMBS into a comfortable retirement.  I get it.  It’s not a choice I’d make, but I can understand.

What I cannot understand, however, is when someone takes a personal decision and tries to use it as a reason to not use a new technology.  Think:  I don’t know MarkLogic, it is new, ergo it is a threat to my personal career plan, and ergo I am opposed to using MarkLogic, prima facie, because it’s not aligned with my personal interests.  That’s not OK.

To give you an idea of how warped this perspective can get (and while this may be urban myth), I recall hearing a story that one time a Federal contractor called a whistle-blower line to report the use of MarkLogic on system instead of Oracle.  All I could think of was Charlton Heston at the end of Soylent Green saying, “I’ve seen it happening … it’s XML … they’re making it out of XML.

The trouble is that these folks exist and they won’t let go.  The result:  when a $630M poorly managed project gets in trouble, they instantly raise and re-raise decisions made about technology with the argument that “it’s non-standard.”

Oracle was non-standard in 1983.  Thirty years later it’s too standard (i.e., part of an oligopoly) and not adapted to the new technical challenges at hand.  All because some bright group of people wanted to try something new, to meet a new challenge, that cost probably a fraction of what Oracle would have charged, the naysayers and Oracle lifers will challenge it endlessly saying it’s “different.”

Yes, it is different.  And that, far as I can tell, was the point.  And if you think that looking at 1% of the costs is the right way to diagnose a struggling $630M project, I’d beg to differ.  Follow the money.


FYI, in researching this post, I found this just-released progress report.

16 responses to “The Pillorying of MarkLogic: Why Selling Disruptive Technology To the Government is Hard and Risky

  1. I appreciate your passion, and I agree with pretty much all your points, except I see the NYT’s reporting as fairly even-handed in this case. They’re quoting a source, and they give MarkLogic an opportunity to respond. There is very little if any editorializing. They wouldn’t be doing their job as reporters on this important story if they didn’t report what the insiders in the project are saying. Also, to put this in perspective, we’re talking about a single graph on the last page of the story. It’s certainly not as if MarkLogic is taking the primary heat here: in fact Oracle appears much more prominently.

    Later in your post you move from questioning the reporting to questioning the attitudes of the contractors who never (it appears) got on board with MarkLogic. You assume this was due to intransigence, but there may have been some other reason. The Times doesn’t take the time to investigate that: perhaps if they did, the vendors would come up short, as you suggest. But would you really want to see even deeper scrutiny into this unhappy shotgun marriage? In my experience, when things go wrong, the blame gets slung around and anyone standing nearby gets some share: sure collateral damage stinks, but it shouldn’t come as a surprise.

    Personally, I can’t help wondering why the decision was made to choose a technology that vendors appear to have rejected, or, conversely, a vendor that didn’t have the skills, knowledge or ambition to learn the technology was chosen.

  2. Pingback: The Pillorying of MarkLogic: Why Selling Disruptive Technology To … | Fred Zimny's Serve4impact

  3. Very insightful in hearing another perspective.Far too easy to blame technology.

  4. Great post Dave!

  5. Hi Dave
    Great article and you think bad in US, UK even worse our Government do not even have “Those who look objectively at requirements and try to figure out which technology can best do the job.” but plenty vested interests in Government and suppliers “Those who look at their own skill sets and view any disruptive technology as a threat” It is called the Innovators Dilemma.

    We have created after 20 years R&D and early adopters created that sought after capability to remove coding from enterprise applications 6GL Adaptive Software. But what a battle…..see interesting recent dialogue here

    Our example over is the Universal Credit where spend heading for £500m! It should have been a fraction (less than 10%) of that with our new approach. Trouble is no one really wants to know and of course real disruptive tech will likely deliver at such dramatic lower cost so not just the Innovators Dilemma credibility is in question!

    It will take strong informed leadership to fix those issues you raise and UK has a long way to go!

  6. Well said, Dave! I’m a fan of MarkLogic and I have not been involved in the project. But as I’ve read reactions to the NYT article and talked with good friends, I’ve been amazed how much negative bias technologists hold against XML, and thereby an XML database. Many don’t want to hear anything more about MarkLogic if they first hear it’s an XML database.

    I can relate to their bias, having worked in 1999 close to a team building a web-scale project using an “object database” which was eventually blamed for scalability issues so the team reluctantly migrated back to Oracle. Later, troubleshooting scalability issues on a different high-traffic SAAS project, I found XML parser Xerces to be the culprit. In resolving the issue, I learned far too much about the overhead of parsing XML. I hate to admit that I even laughed in the face of a non-technical manager when he cornered me just to tell me I should love XML because it “makes everything faster”.

    Yet thankfully I was able to set aside my bias when a trusted friend recommended I look closer at MarkLogic. The technology I found simply astounded me in a good way. I get all the benefits of an object database (now we say “document database”) with incredible support for well-defined XML data syntax, all built on a search engine, plus a whole lot more. But I’ll step off my soap box about how awesome MarkLogic is. My point is I understand the bias against XML and alternative databases. We’ve all seen both used inappropriately, just as we’ve seen RDBMS tech used inappropriately. What I don’t get is how far some will ride their bias, without double-checking their assumptions / ignorance. From where I stand, those willing to stand up against MarkLogic technology without understanding it just make themselves look silly.

  7. Pingback: The War Over Open Platform Services Is Just Getting Started | DIGIZENS

  8. Reblogged this on Labyrinth, Ideas, and Wanderings and commented:
    Ok, to my usual writing friends this will seem like a foreign language, but perhaps this article will shed some light on just why the government website for the health insurance exchanges had such difficult problems. One comment I found interesting is Mr. Kellogg ‘ s statement that he hadn’t counted on “the politics” in dealing with a government contract. I thought that’s what government IS: politics. I wouldn’t expect it to be any different. Frankly that’s why I have avoided taking government jobs when I have considered doing something different with my career. Job stability in government – based settings is a thing of the past. Big government is no longer the savior of the people. Put that poison goblet down.

  9. Pingback: The War Over Open Platform Services Is Just Getting Started «

  10. Pingback: Four ways to improve federal software development | TechTuesdayBlog

  11. Pingback: In Defense of MarkLogic : Stephen E. Arnold @ Beyond Search

  12. I agree that you have to pick the right solution for the right tool. Picking a disruptive technology where there is very little in-house knowledge or expertise, has a small on-line community, the error messages are not documented, very little monitoring or views into what threads are consuming most resources, the statement they are running, or what resource they are waiting on, an architecture that spans across multiple hosts (into the 100s) that compounds complexity in managing and troubleshooting, and a vendor whose response is a contract for their professional services because it is large source of their revenue is not the right choice here.

    MarkLogic is a great solution for something that you aren’t betting your job or company’s future on.

    Great post, by the way. You completely nailed the problem – it wasn’t MarkLogic, it was the person who was in charge of the project and decided to use MarkLogic.

  13. Now that the dust has cleared and marklogic seems to be doing ok. I do not read to many post from the detractors who blamed this on the selection of MarkLogic. Oregon who chose a 100% Oracle solution was unable to deliver a product at all. But this is never stated in any blog.

  14. Pingback: Understanding the Capabilities within | adventuresinhealthinformatics

  15. Pingback: Understanding the Vision of | adventuresinhealthinformatics

  16. Pingback: The Major Stakeholders of | adventuresinhealthinformatics

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.