Category Archives: XQuery

The 20th Century Called. It Wants Its Relational Database Back.

I saw this piece of creative the other day for a tradeshow ad and loved it.  Remember, Ted Codd invented the relational database in 1970 with his paper “A Relational Model for Shared Data Banks.”  This PDF of the classic looks about as old as the ad.  (Do PDFs age?)  Enjoy!

Six Thoughts on The NoSQL Movement

We are in the middle of one of our periodic analyst tours at MarkLogic, where we meet about 50 top software industry analysts focused in areas like enterprise search, enterprise content management, and database management systems.  The NoSQL movement was one of four key topics we are covering, and while I’d expected some lively discussions about it, most of the time we have found ourselves educating people about NoSQL.

In this post, I’ll share the six key points we’re making about NoSQL on the tour.

Our first point is that NoSQL systems come in many flavors and it’s not just about key/value stores.  These flavors include:

  • Key/value stores (e.g., Hadoop)
  • Document databases (e.g., MarkLogic, CouchDB)
  • Graph databases (e.g., AllegroGraph)
  • Distributed caching systems (e.g., Memcached)

Our second point is that NoSQL is part of a broader trend in database systems:  specialization.  The jack-of-all-trades relational database (e.g., Oracle, DB2) works reasonably well for a broad range of applications — but it is a master of none.  For any specific application, you can design a specialized DBMS that will outperform Oracle by 10 to 1000 times.  Specialization represents, in aggregate, the biggest threat to the big-three DBMS oligopolists.  Examples of specialized DBMSs include:

  • Streambase, Skyler:  real-time stream processing
  • MarkLogic:  semi-structured data
  • Vertica, Greenplum:  mid-range data warehousing
  • Aster:  large-scale (aka “big data”) analytic data warehousing
  • VoltDB:  high volume transaction processing
  • MATLAB:  scientific data management

Our third point is that NoSQL is largely orthogonal to specialization.  There are specialized NoSQL databases (e.g., MarkLogic) and there are specialized SQL databases (e.g., Aster, Volt).  The only case where I think there are zero examples is general-purpose NoSQL systems.  While I’m sure many of the NoSQL crowd would argue that their systems can do everything, is anyone *really* going to run general ledger or opportunity management on Hadoop?   I don’t think so.

Our fourth point is that NoSQL isn’t about open source.  The software-wants-to-be-free crowd wants to build open source into the definition of NoSQL and I believe that is both incorrect and a mistake.  It’s incorrect because systems like MarkLogic (which uses an XML data model and XQuery) are indisputably NoSQL.  And it’s a mistake because technology movements should be about technology, not business models.  (The open source NoSQL gang can solve its problem simply by affiliating with both the NoSQL technology movement and the open source business model movements.)

As CEO of a company that’s invested a lot of energy in supporting standards, our fifth point was that, rather ironically, most open source NoSQL systems have proprietary interfaces.  People shouldn’t confuse “can access the source code” with “can write applications that call standard interfaces” and ergo can swap components easily.   If you take offense at the word proprietary, that’s fine.  You can call them unique instead.  But the point is an application written on Cassandra is not practically moved to Couch, regardless of whether you can access the source code both Couch and Cassandra.

Our sixth point is that we think MarkLogic provides a best-of-both-worlds option between open source NoSQL systems and traditional DBMSs.  Like open source NoSQL systems, MarkLogic provides shared-nothing clustering on inexpensive hardware, superior support for unstructured data, document-orientation, and high-performance.  But like traditional databases, MarkLogic speaks a high-level query language, implements industry standards, and is commercial-grade, supported software.  This means that customers can scale applications on inexpensive computers and storage, avoid the pains of normalization and joins, have systems that run fast, can be implemented by normal database programmers, and feel safe that their applications are built via a standard query language (XQuery) that is supported by scores of vendors.

Yes, Virginia, MarkLogic is a NoSQL System

The other day I noticed a taxonomy used on one of the NoSQL Database blogs that went like this:

Types of NoSQL systems

  • Core NoSQL Systems
    • Wide column stores
    • Document stores
    • Key-value / tuple stores
    • Eventually consistent key-value stores
    • Graph databases
  • Soft NoSQL Systems (not the original intention …)
    • Object databases
    • Grid database solutions
    • XML databases
    • Other NoSQL-related databases

I, perhaps obviously, take some umbrage at having MarkLogic (acceptably classified as an XML database) being declared “soft NoSQL.”  In this post I’ll explain why.

Who decided that being open source was a requirement to be real NoSQL system?  More importantly, who gets to decide?  NoSQL – like the Tea Party – is a grass-roots, effectively leaderless movement towards relational database alternatives.  Anyone arguing original intent of the founders is misguided because there is no small group of clearly identified founders to ask.  In reality, all you can correctly argue is what you think was the intent of the initial NoSQL developers and early adopters, or — perhaps more customarily — why you were drawn to them yourself, disguised or confused as original founder intent.

As mentioned here, movements often appear homogeneous when they are indeed heterogeneous.  What looks like a long line of demonstrators protesting a single cause is in fact a rugby scrum of different groups pushing in only generally aligned directions.  For example, for each of the following potential motivations, I am certain that I can find some set of NoSQL advocates that are motivated by it:

  • Anger at Oracle’s heavy-handed licensing policies
  • The need to store unstructured or semi-structured data that doesn’t fit well into relations
  • The impedance mismatch with relational databases
  • A need and/or desire to use open source
  • An attempt to reduce total cost
  • A desire to land at a different point in the Brewer CAP Theorem triangle of consistency, availability, and partition tolerance
  • Coolness / wannabe-ism, as in, I want to be like Google or Facebook

(Since this was a source of confusion in prior posts, note that this is not to claim the inverse:  that all NoSQL advocates are motivated by all of the possible motivations.)

I’d like to advocate a simple idea:  that NoSQL means NoSQL.  That a NoSQL system is defined as:

A structured storage system that is not based on relational database technology and does not use SQL as its primary query language

In short, my proposed definition means that NoSQL (broadly) = NoSQL (literally) + NoRelational.  In short:  relational database alternatives.  It does not mean:

  • NoDBMS.  We should not take NoSQL to exclude systems we would traditionally define as DBMSs.  For example, supporting ACID transactions or supporting a non-SQL query language (e.g., XQuery) should not be exclusion criteria for NoSQL.
  • NoCommercialSoftware.  While many of the flagship NoSQL projects (e.g., Hadoop, CouchDB) are open source projects, that should be not a defining criterion.  NoSQL should be a technological, not a delivery- or business-model, classification.  Technology and delivery model are orthogonal dimensions.   We should be able to speak of traditionally licensed, open source licensed, and cloud-hosted NoSQL systems if for no other reason than understanding the nuances of the various business/delivery models is a major task unto itself.  Do you mean open source or open core?  Is it open source or faux-pen source?  Under which open source license?  How should I think of a hosted subscription service that is a based on or a derivative of an open source project?

Recently, I’ve heard a piece of backpeddling that I’ve found rather irritating:  that NoSQL was never intended to mean “no SQL,” it was actually intended to mean “not only SQL.”  Frankly, this strikes me as hogwash:  uh oh, I’m afraid that people are seeing us as disruptors and it’s probably easier to penetrate the enterprise as complementary, not competitive, so let’s turn what was a direct assault into a flanking attack.

To me, it’s simple:  NoSQL means NoSQL.  No SQL query language and no relational database management system.  Yes, it’s disruptive and — by some measures — “crazy talk” but no, we shouldn’t hide because there are lots of perfectly valid (and now socially acceptable) reasons to want to differ from the relational status quo.

In effect, my definition of NoSQL is relational database alternative.  Such options include both alternative databases (e.g., MarkLogic) and database alternatives (e.g., key/value stores).  This, of course, then cuts at your definition of database management system where I (for now at least) still require the support of a query language and the option to have ACID transactions.

By the way, I understand the desire to exclude various bandwagon-jumpers from the NoSQL cause.  Like most, I have no interest in including thrice-reborn object databases in the discussion, but if the cost of excluding them is excluding systems like MarkLogic then I think that cost is too high.  Many people contemplating the top-of-mind NoSQL systems (e.g., Hadoop) could be better served using MarkLogic which addresses many typical NoSQL concerns, including:

  • Vast scale
  • High performance
  • Highly parallel shared-nothing clusters
  • Support for unstructured and semi-structured data

All with all the pros (and cons) of being a commercial software package and without requiring reduced consistency:  losing a few Tweets won’t kill Twitter, but losing a few articles, records, or individuals might well kill a patient, bank, or counter-terrorism agency.  BASE is fine for some; many others still need ACID.  Michael Stonebraker has some further points on this idea in this CACM post.

I’d like to suggest that we should combine the ideas in this post with the ideas in my prior one, Classifying Database Management Systems.  That post says the correct way to classify DBMSs is by their native modeling element (e.g., table, class, hypercube).  This post says that NoSQL is semi-orthogonal – i.e., I can imagine a table-oriented database that doesn’t use SQL as its query language, but I doubt that any exist.  Applying my various rules, the combined posts say that:

  • Aster is a SQL database optimized for analytics on big data
  • MarkLogic is an XML [document] database optimized for large quantities of semi-structured information and a NoSQL system
  • CouchDB is a document database and a NoSQL system
  • Reddis is a key/value store and a NoSQL system
  • VoltDB is a SQL database optimized to solve one of the two core problems that NoSQL systems are built for (i.e., high-volume simple processing)

Finally, I’d conclude that even with these rules I have trouble classifying MarkLogic because of multiple inheritance:  MarkLogic is both a document database and an XML database, it is difficult to pick one over the other, and I there certainly are non-document-oriented XML database systems.   Similar issues exist with classifying the various hybrids of document databases and key/value stores.  So while I may have more work to do on building an overall taxonomy, I am absolutely sure about one thing:  MarkLogic is a NoSQL system.


* The “Yes, Virginia” phrase comes from a 1897 story in the New York Sun.  For more, see here.

My Thoughts on the NoSQL Database "Tea Party" Post

Without a doubt, the most controversial post I’ve written on this blog was last month’s The Database Tea Party: The NoSQL Movement.  I know this both from the comment stream, but also from the volume and tenor of emails I received from friends and colleagues over the past few weeks.

The first thing I learned is that standing in the middle is a great way to get attacked from all sides.

  • My database buddies blasted me, treating me like a reckless turncoat:   how dare you endorse these BASE people?  (Humorous double entendre intended.)
  • The NoSQL folks blasted me, generally misunderstanding the protest march metaphor I was using (see below).
  • The above-it-all crowd blasted me for oversimplifying the issue, suggesting that I was endorsing simplistic views such as it’s about batch vs. online or it’s about scaling vs. not scaling.

All of which, of course, only confirmed the religious nature of the movement and that was indeed a movement, regardless of whether any given participant identified himself as such.

(Several folks also blasted me for using the Tea Party Movement as a metaphor.  Let me clarify that this is not a political blog so I won’t debate politics.  I intended the metaphor to cover only concepts like “rebellion” and “grass roots,” both of which I do believe apply to NoSQL.)

First, I’ll clarify the protest march metaphor which was easily the most misunderstood aspect of the post.   Let me share my first rule of protests:  not everyone is there for the same reason.  Some people are there for the stated cause, which I’ll call cause A.  But others participate on behalf of a group; their signs will say “Group 1’s for Cause A.”  Others are there for cause B which lacks enough support to generate its own march, so they tag along with signs saying “Cause A and Cause B.”  If you’ve ever been at a rally, you’ve invariably winced when speakers attempted to hijack the agenda, turning to some personal cause, all while pretending to speak on behalf of the group.

It was this sense of chaos and disorder that I was trying to portray.  When I made the list of reasons why I thought people were on the NoSQL march, it was neither to say that I agreed with them or that all people were there for all reasons.  I was doing the equivalent of asking protesters on the  UC Berkeley Anti-Grenada March why they were there.  To which the replies might have been:

  • To protest the Grenada invasion (cause A)
  • To remind people about [insert group here] rights, all while protesting against Grenada
  • To protest about UC budget cuts, all while protesting about something the Government did, which I’m pretty sure was bad.
  • Because I go to every march; what’s this one about?
  • Because I was at Top Dog when it went by and had nothing better to do

So hopefully, the intent of my NoSQL-reasons-list is now clear.   Some folks are there because they don’t want ACID transactions.  Some folks are there because they are dealing with Internet scale.  Some folks are there because they hate the SQL impedance mismatch.  Some folks are there because they’re tired of paying oligopoly prices to Oracle.   (I particularly liked the comment that said Oracle was free because they had an enterprise license that most certainly wasn’t, ignoring the possibility that recent enterprise/agency directives to look at open source could result directly from the size of the last Oracle check.)

And yes, some folks are there because it’s cool and they want to be like Twitter, Google, and Facebook, but getting them to admit that is a virtual impossibility.

One irony is that I actually agree with one of the fiercest commenters:

A very interesting write-up with one little oversight: you’re wrong.

I am part of a large program to write a NoSQL database for military applications. [It’s not about …] It’s [about]  the fact that RDBMSs are built in a different space in the CAP trades.

Google, Amazon, Facebook, and DARPA all recognized that when you scale systems large enough, you can never put enough iron in one place to get the job done (and you wouldn’t want to, to prevent a single point of failure). Once you accept that you have a distributed system, you need to give up consistency or availability, which the fundamental transactionality of traditional RDBMSs cannot abide. Based on the realization that something fundamentally different needed to be built, a lot of very smart people tackled the problem in a variety of different ways, making different trades along the way. […]

So – the NoSQL databases are a pragmatic response to growing scale of databases and the falling prices of commodity hardware. It’s not a noble counterculture movement (although it does attract the sort that have a great deal of mental flexibility), it’s just a way to get business done cheaper.

To respond to the commenter:

  • Thank you for the clear definition of why you moved to NoSQL.
  • Your comment was picked up by the Otaku blog in an post called NoSQL Explained Correctly (Finally), so congrats and I’m glad I could help facilitate “the conversation.”
  • Disorder and chaos is what I was trying to portray in the protest march metaphor, not hippies or nobility
  • You are clearly using NoSQL for two of the reasons on my list:  scalability and bloatware (i.e., perhaps not the best word choice, but the idea was undesired, included functionality — e.g., ACID transactions)
  • You did exactly what I said to do:  consider all alternatives and do what’s right for your business
  • So, why are we disagreeing again?

I think some people didn’t like my putting “coolness” on the table as a factor and the notion of a “movement.”  I believe those are both very real and ironically those who disagreed with me loudest were effectively screaming:  it’s not a movement and I’m not doing it to be cool; I’m doing it because it’s right for my business.  If so, great.  But why does it hit such a nerve?

In the end, when it comes to NoSQL I am trying to:

  • Provide an overview of why I think people are considering and/or using NoSQL solutions
  • Provide good background references and readings (see bottom of my first post)
  • Remind mangers to keep an eye out for the “bad reasons” to go NoSQL — i.e,. coolness and Google wannabeism
  • Remind people not to confuse NoSQL with NoDatabase.  Special-purpose databases (e.g., MarkLogic) are optimized for specific applications (e.g., semi-structured data) and handle them far better than a general-purpose RDBMS.  So in your haste to move off Oracle, don’t advance directly to an open source key-value store; there might be alternative DBMSs that meet your needs more effectively.
  • Remind people not to confuse NoSQL with NoCommercialSoftware.  While people seem to dislike when I say it, the RDBMS market is an oligopoly and the big vendors’ pricing, margins, and heavy-handed customer relationships are all consistent with that market structure.  But you can find other classes of commercial software where the vendors are hungrier and more customer centric.

Fun at the Mark Logic Company Picnic

Like many companies, we have an annual summer event. Since many Mark Logicians have young children, we bias the entertainment a bit in that direction. This year we had a face-painter, a bubble-blower, and a truly amazing balloon-twister.

During the event, two of our marketers cut the face-painting line (making a few kids cry in the process) and put the face-painter to good marketing use by, well, body-painting some key Mark Logic messaging on themselves: Mark Logic — The Best Place to Put Your XML.

(Big, bad a** fast, XQuery is on the arms.)

Gosh, we have so much fun around here at times it scares me.

Norm Walsh Making The Case For XQuery

Mark Logic‘s Norm Walsh recently wrote an article for the Data Conversion Labs website, entitled Making the Case for XQuery. Excerpt:

But now that you have an XML repository, what are you going to do with it? What and how may you deploy it? Simply having large masses of XML converted data doesn’t necessarily mean that the data in this form is even useful.

Enter XQuery.

Norm then explains the problem with using other languages with XML documentbases

The problem with other programming languages isn’t that they aren’t able to process XML, it’s that they aren’t able to process XML efficiently. Data has to be converted from XML to the language’s native data structures. Once converted, it must be manipulated with functions that don’t understand the underlying model and are, consequently, not always a good fit. This “impedance mismatch” causes confusion and can introduce errors. Finally, the programming language structures have to be converted back into XML. Each of these steps is tedious, time consuming, and introduces the possibility of errors. In a sophisticated application, this process may have to occur several times for each XML resource.

If all this looks interesting, the full article is here.

Related articles by Zemanta

Gartner Names "Specialized Systems" A Top 10 Strategic Technology

Leading IT analyst firm Gartner has named “specialized systems” to its list of top 10 strategic technologies for 2009. While I’m sure Gartner wasn’t thinking specifically of Mark Logic (for, among other reasons, that we’ve not spoken with David Cearley though I do know him from my Business Objects days), I would indeed argue that Mark Logic fits perfectly into this trend.

Here’s what Gartner says about specialized systems:

Specialized Systems. Appliances have been used to accomplish IT purposes, but only with a few classes of function have appliances prevailed. Heterogeneous systems are an emerging trend in high-performance computing to address the requirements of the most demanding workloads, and this approach will eventually reach the general-purpose computing market. Heterogeneous systems are also specialized systems with the same single-purpose imitations of appliances, but the heterogeneous system is a server system into which the owner installs software to accomplish its function.

While this is a generalized description, the point is clear: for high-performance computing, you will increasingly partition your workload amongst a heterogeneous network of servers each designed and optimized for a specific task. For MarkLogic Server, that task is high-performance XQuery evaluation against large XML databases, documentbases, and/or contentbases.

I’d also say that this argument is similar to one that Mike Stonebraker makes: that as you partition your workload against various, specialized (database) servers (e.g., OLTP, data warehousing, stream processing, XML processing, scientific data processing) you will find that, by elimination, there is no apparent need for a general-purpose database. That is, that every purpose a DBMS serves is a special purpose and we will therefore soon see the end of the era dominated by the general-purpose DBMS.

By the way, I’d also argue that Mark Logic has a role in one of Gartner’s other top 10 trends, web-oriented architectures.

Web-Oriented Architectures. The Internet is arguably the best example of an agile, interoperable and scalable service-oriented environment in existence. This level of flexibility is achieved because of key design principles inherent in the Internet/Web approach, as well as the emergence of Web-centric technologies and standards that promote these principles. The use of Web-centric models to build global-class solutions cannot address the full breadth of enterprise computing needs. However, Gartner expects that continued evolution of the Web-centric approach will enable its use in an ever-broadening set of enterprise solutions during the next five years.

As I’ve said here before, once a customer starts to use MarkLogic as a platform / repository / search engine for their XML, they soon realize that it’s easier to write web applications in a pure top-to-bottom XML fashion than in the dual mapping from an XML-oriented browser to a object-oriented Java layer to a table-oriented (relational) DBMS. That’s the subject of a different post. If you’re interested in top-to-bottom XML, then go here.

Gartner’s top 10 list of strategic technologies for 2009 is here.

Related articles by Zemanta
Reblog this post [with Zemanta]