I suppose it’s not surprising that on the journey to find my ideal next gig that I’ve seen a lot of next-generation business intelligence (BI) companies. Because I’ve thus had the chance to immerse myself in the BI startup world, I thought I’d share a quick glimpse of what’s presumably the BI future.
Because some of the companies I’ve seen are still stealth, I’m not going to name any early-stage names, but simply provide a list of common traits of next-generation BI companies.
Traits of next-generation BI:
- In memory, columnar, and compressed. Most solutions rely on the fact that the source data for most problems can now fit in memory, typically using a columnar and compressed format. Some solutions are even able to perform work on the data without first decompressing it.
- Fast. The dream of BI — particularly for interactive analysis tools — has always been “speed of thought” analysis. Thanks to the above point and thanks to additional performance optimizations (e.g., to expoit CPU cache locality), this dream is becoming a reality.
- Directly connected. Next-generation BI tools generally connect directly to the underlying source databases (and/or the Internet) to capture data. This means they must also have basic data integration capabilities both so they properly align data from different systems and dynamically refresh it.
- Schema-free. In order to accomodate semi-structured information and to be able integrate information from different sytems, next-generation BI does not require the up-front definition of a schema. Instead, relationships among data (e.g., hierarchy) are discovered dynamically.
- Beautiful. While this is best exemplified by Tableau (where visualization is the principal focus) next-generation BI tools generally provide beautiful visualizations that are more powerful than the basic report and bar chart. (Note that I named a name here because I consider Tableau mid-stage, not early-stage.)
- Mobile. Next-generation BI tools typically assume a brower-based client and often the need to create device-specific clients (e.g., a native iPad app) to supplement it. Some companies focus exclusively on mobile BI.
- Neutral. Next-generation BI tools exploit the fact that a multi-billion dollar vacuum was created in the market when the BI leaders were consolidated and became units of IBM (e.g., Cognos) or SAP (e.g., BusinessObjects).
In many ways, next-generation BI takes us full circle back to the days of Cognos PowerPlay and its desktop-resident PowerCube (i.e., hypercube) — except that the cube is now virtual, schema-free, of effectively unlimited size, and contains no precalcuated aggregates. But like that era, the cube in many ways obviates the data warehouse infrastructure underneath it. After all, if you can fit your entire data set in memory and dynamically calculate the answer to any question at high speed, then why do you need a data warehouse full of precalculated aggregates again?
The answer is “you do” for many cases (e.g., history, data cleansing) — but certainly not for all of them. I thus see a “middle squeeze” on the data warehouse market in the future.
- For most applications of normal size and analytic complexity, people will use next-generation BI on top of raw data sources, unless they have very messy data or a need for extensive history.
- For large applications (i.e., big data) and/or high analytic complexity, people will use advanced analytic platforms (e.g, Aster Data). This, of course, begs the question whether anyone is working on BI tools that exploit and optimize the new, high-end analytic engines and the answer to that question is happily “yes” as well.