Types of NoSQL systems

• Core NoSQL Systems
  • Wide column stores
  • Document stores
  • Key-value / tuple stores
  • Eventually consistent key-value stores
  • Graph databases

• Soft NoSQL Systems (not the original intention …)
  • Object databases
  • Grid database solutions
  • XML databases
  • Other NoSQL-related databases

I, perhaps obviously, take some umbrage at having MarkLogic (acceptably classified as an XML database) being declared “soft NoSQL.”  In this post I’ll explain why.

Who decided that being open source was a requirement to be real NoSQL system?  More importantly, who gets to decide?  NoSQL – like the Tea Party – is a grass-roots, effectively leaderless movement towards relational database alternatives.  Anyone arguing original intent of the founders is misguided because there is no small group of clearly identified founders to ask.  In reality, all you can correctly argue is what you think was the intent of the initial NoSQL developers and early adopters, or — perhaps more customarily — why you were drawn to them yourself, disguised or confused as original founder intent.

As mentioned here, movements often appear homogeneous when they are indeed heterogeneous.  What looks like a long line of demonstrators protesting a single cause is in fact a rugby scrum of different groups pushing in only generally aligned directions.  For example, for each of the following potential motivations, I am certain that I can find some set of NoSQL advocates that are motivated by it:

• Anger at Oracle’s heavy-handed licensing policies
• The need to store unstructured or semi-structured data that doesn’t fit well into relations
• The impedance mismatch with relational databases
• A need and/or desire to use open source
• An attempt to reduce total cost
• A desire to land at a different point in the Brewer CAP Theorem triangle of consistency, availability, and partition tolerance
• Coolness / wannabe-ism, as in, I want to be like Google or Facebook

(Since this was a source of confusion in prior posts, note that this is not to claim the inverse:  that all NoSQL advocates are motivated by all of the possible motivations.)

I’d like to advocate a simple idea:  that NoSQL means NoSQL.  That a NoSQL system is defined as:

A structured storage system that is not based on relational database technology and does not use SQL as its primary query language

In short, my proposed definition means that NoSQL (broadly) = NoSQL (literally) + NoRelational.  In short:  relational database alternatives.  It does not mean:

• NoDBMS.  We should not take NoSQL to exclude systems we would traditionally define as DBMSs.  For example, supporting ACID transactions or supporting a non-SQL query language (e.g., XQuery) should not be exclusion criteria for NoSQL.

• NoCommercialSoftware.  While many of the flagship NoSQL projects (e.g., Hadoop, CouchDB) are open source projects, that should be not a defining criterion.  NoSQL should be a technological, not a delivery- or business-model, classification.  Technology and delivery model are orthogonal dimensions.   We should be able to speak of traditionally licensed, open source licensed, and cloud-hosted NoSQL systems if for no other reason than understanding the nuances of the various business/delivery models is a major task unto itself.  Do you mean open source or open core?  Is it open source or faux-pen source?  Under which open source license?  How should I think of a hosted subscription service that is a based on or a derivative of an open source project?

Recently, I’ve heard a piece of backpeddling that I’ve found rather irritating:  that NoSQL was never intended to mean “no SQL,” it was actually intended to mean “not only SQL.”  Frankly, this strikes me as hogwash:  uh oh, I’m afraid that people are seeing us as disruptors and it’s probably easier to penetrate the enterprise as complementary, not competitive, so let’s turn what was a direct assault into a flanking attack.

To me, it’s simple:  NoSQL means NoSQL.  No SQL query language and no relational database management system.  Yes, it’s disruptive and — by some measures — “crazy talk” but no, we shouldn’t hide because there are lots of perfectly valid (and now socially acceptable) reasons to want to differ from the relational status quo.

In effect, my definition of NoSQL is relational database alternative.  Such options include both alternative databases (e.g., MarkLogic) and database alternatives (e.g., key/value stores).  This, of course, then cuts at your definition of database management system where I (for now at least) still require the support of a query language and the option to have ACID transactions.

By the way, I understand the desire to exclude various bandwagon-jumpers from the NoSQL cause.  Like most, I have no interest in including thrice-reborn object databases in the discussion, but if the cost of excluding them is excluding systems like MarkLogic then I think that cost is too high.  Many people contemplating the top-of-mind NoSQL systems (e.g., Hadoop) could be better served using MarkLogic which addresses many typical NoSQL concerns, including:

• Vast scale
• High performance
• Highly parallel shared-nothing clusters
• Support for unstructured and semi-structured data

All with all the pros (and cons) of being a commercial software package and without requiring reduced consistency:  losing a few Tweets won’t kill Twitter, but losing a few articles, records, or individuals might well kill a patient, bank, or counter-terrorism agency.  BASE is fine for some; many others still need ACID.  Michael Stonebraker has some further points on this idea in this CACM post.

I’d like to suggest that we should combine the ideas in this post with the ideas in my prior one, Classifying Database Management Systems.  That post says the correct way to classify DBMSs is by their native modeling element (e.g., table, class, hypercube).  This post says that NoSQL is semi-orthogonal – i.e., I can imagine a table-oriented database that doesn’t use SQL as its query language, but I doubt that any exist.  Applying my various rules, the combined posts say that:

• Aster is a SQL database optimized for analytics on big data
• MarkLogic is an XML [document] database optimized for large quantities of semi-structured information and a NoSQL system
• CouchDB is a document database and a NoSQL system
• Reddis is a key/value store and a NoSQL system
• VoltDB is a SQL database optimized to solve one of the two core problems that NoSQL systems are built for (i.e., high-volume simple processing)

Finally, I’d conclude that even with these rules I have trouble classifying MarkLogic because of multiple inheritance:  MarkLogic is both a document database and an XML database, it is difficult to pick one over the other, and I there certainly are non-document-oriented XML database systems.   Similar issues exist with classifying the various hybrids of document databases and key/value stores.  So while I may have more work to do on building an overall taxonomy, I am absolutely sure about one thing:  MarkLogic is a NoSQL system.

—
* The “Yes, Virginia” phrase comes from a 1897 story in the New York Sun.  For more, see here.

Excerpt:

Moreover, the code line from all of the major vendors is quite elderly, in all cases dating from the 1980s. Hence, the major vendors sell software that is a quarter century old, and has been extended and morphed to meet today’s needs. In my opinion, these legacy systems are at the end of their useful life. They deserve to be sent to the “home for tired software.”

His key argument is all about performance: in any given use-case, Stonebraker thinks RDBMSs can be beaten by about a factor of 50.

• In data warehousing he says a column store wins by 50x

• In OLTP he says a memory-resident DBMS wins by 50x

• For scientific data, he says a DBMS specialized for the job can win by 50x

• For RDF, he says column stores do a reasonable job and is confident that specialized RDF triple stores will do better, i.e., 50x or more. (I’d add that at MarkLogic we think we do a reasonable job as well.)

• For text, he points out that no major search engine uses a relational database so they didn’t even qualify for consideration.

• For XML, he cites a private report I sent him a while back done for one of our customers comparing MarkLogic performance to a relational DBMS. When on “our turf,” we usually win by no less than 10x and sometimes 100x or more. Sometimes, queries are not even processable in an RDBMS and/or need to be hand-optimized and hand-joined between a DBMS and a search engine.

He reduces to three cases how special-purpose DBMS vendors get their advantage:

• A non-relational data model
• A different implementation of tables
• A different implementation of transactions

We’re in the first category, using XML as our data model instead of a table. It’s a great post. Check it out and check out the cited references as well.

First, a nit. When I was a lad, “proprietary” didn’t mean “closed source“, it meant proprietary (i.e., vendor controlled) interface. For example, Ingres originally spoke a query language called Quel. SQL then emerged as the standard and any DBMS that spoke a language other than ANSI standard SQL was deemed proprietary. While I know that some people in the open source community view the opposite of “open source” as “proprietary,” I think that’s a misnomer. I think the correct antonym is closed source.

First, I think Allan makes an excellent point about stagnation:

By the turn of the millenium, relational databases had already pretty much met the essential requirements of end users, and proprietary database companies were either pointing their vaccuum cleaners toward other interesting money piles, or losing the plot entirely and sailing off the edge of the world. Today, database releases continue to tout new features, but they’re frosting on the cake rather than essentials. No-one issues a tender for a database unless they have unusual requirements. No-one loses their job because they chose the wrong database. And it’s been that way for years.

As a general rule I am shocked by the lack of innovation returned by the R&D budgets of most technology companies. As I mentioned yesterday, despite billions of R&D investment, Google has yet to come up with another big business. And what does Microsoft get for the billions they spend each year on R&D? An incompatible version of Office with irritating “ribbons” that takes four years to make.

Silicon Valley startups create new categories with $10s of millions in venture capital. It seems that once they become “real companies” they forget how to innovate at all, let alone on a shoestring.

Specifically in the DBMS market, I think the lack of innovation — enabled by the oligopolistic structure of the market — creates a soft underbelly for focused, innovative companies to carve our niches. (And remember “niches” of $10B market can be pretty big.)

Allan goes on to do some interesting pricing analysis, and then poses the question:

Why, then, is proprietary database software becoming more expensive while everything else reduces in price? End users normally expect to benefit from the cost savings resulting from improvements in technology. I am writing this blog, for example, on an affordable computer that would easily outperform expensive commercial systems from just 10 years ago.

It seems difficult to resist the conclusion that proprietary database companies have managed to redirect a good chunk of these savings away from end users and into their own coffers. Successful as this strategy has been, though, it could ultimately backfire. The more expensive proprietary databases become, the more attractive lower cost alternatives appear.

I think the short answer to his question is (1) the market is an oligopoly and (2) there is a lot of inertia when it comes to database management systems. So change will happen, but it will happen slowly. And, ironically, the force that drives the market change will be overpricing on the leaders’ part. Were RDBMSs not so expensive, there would be less impetus to move to open source.

Now, the RDBMS vendors probably argue they should “milk” the market until the real threat emerges and then “wave a wand” to reduce price, but that is a risky strategy because they could very easily wave the wand too late, which is what I think they are doing.

The only point I think Alan misses in his analysis is that some powerful vendors like SAP and EMC don’t like the fact that their applications run on top of lower-level DBMS technologies from competitors. For example, SAP has been trying to get itself off Oracle for about a decade, and I’m told they fund developers to work on MySQL towards that end. I know that EMC/Documentum is not comfortable that the vendors who provide the DBMSs they run on are all now challenging them in content management (e.g., Oracle/Stellent, IBM/FileNet, Microsoft SharePoint).

He then speculates on what he thinks will happen going forward:

My vote for the Strategy Most Likely To Succeed is a tie between Revenue Pull-Through and Reduce Prices. Oracle is arguably becoming the most successful proponent of the pull-through strategy. Oracle wants to supply you with a full software stack, including an OS, virtualization software, a broad range of middleware, a database, and end user applications. The largest component of Oracle’s revenue currently still comes from database licenses, but the company is working hard to reduce that dependency. Until that happens, reducing prices across the board will be challenging for Oracle. If Oracle succeeds with a pull-through strategy, it doesn’t mean that OSDBs will fail, of course. It simply means that Oracle is less likely to sustain major damage from their success.

He concludes:

Are proprietary databases doomed, then? Not at all. Even if proprietary database companies pull no surprises, they won’t fade away anytime soon … Make no mistake, though, open source databases are coming. For established companies it’s more likely to be an evolution than a revolution.

I believe there are two major trends in the DBMS market today: (1) open-source chipping away at the closed-source oligopoly, and (2) special-purpose DBMSs innovating and carving out niches in the soft underbelly. I actually think point 1 provides powerful “air cover” for vendors pursuing strategy 2, because point 1 is a direct attack on the existing business.

The article begins with:

Most databases are based on technology that originated 30 years ago. But change is in the air.

A mob of start-ups have been developing variants of the software, which provides the equivalent of filing cabinets for corporate information. Customers say the offerings are generating faster answers to questions that require sifting through huge volumes of business information.

Established suppliers aren’t conceding much to the newcomers, but industry executives agree the pace of progress is accelerating.

“The database market is going to be an exciting place to be in the next decade,” said Michael Stonebraker, an adjunct professor at the Massachusetts Institute of Technology …

My favorite excerpt includes this quote from chief database guru at Gartner, Donald Feinberg (whose actual title is vice president and distinguished analyst).

Some predict specialized products will find a niche. “One kind of database is not going to suit all of the different applications we are coming up with,” said Donald Feinberg, an analyst at market researcher Gartner Inc.

I couldn’t say it any better myself. And — whack — think about this. You have the head database analyst at Gartner saying that one kind of database isn’t going to meet all needs. That’s big. And he’s right.

From the paper’s summary:

“In the last quarter of a century, there has been a dramatic shift in:

1. DBMS markets: from business data processing to a collection of markets, with varying requirements
2. Necessary features: new requirements include shared nothing support and high availability
3. Technology: large main memories, the possibility of hot standbys, and the web change most everything

The result is:

1. The predicted demise of “one size fits all”
2. The inappropriateness of current relational implementations for any segment of the market
3. The necessity of rethinking both data models and query languages for the specialized engines, which we expect to be dominant in the various vertical markets”

As you know, I’m a big believer in the special-purpose DBMS meme. Any database historian knows what Codd was thinking, and more importantly — what he wasn’t — when he designed the relational model. Again, from the paper:

“Ted Codd’s idea of normalizing data into flat tables has served our community well over the subsequent 30 years. However, there are now other markets, whose needs must be considered. These include data warehouses, web-oriented search, real-time analytics, and semi-structured data markets.”

The complete paper can be found here.