I suppose it’s not surprising that on the journey to find my ideal next gig that I’ve seen a lot of next-generation business intelligence (BI) companies. Because I’ve thus had the chance to immerse myself in the BI startup world, I thought I’d share a quick glimpse of what’s presumably the BI future.
Because some of the companies I’ve seen are still stealth, I’m not going to name any early-stage names, but simply provide a list of common traits of next-generation BI companies.
Traits of next-generation BI:
- In memory, columnar, and compressed. Most solutions rely on the fact that the source data for most problems can now fit in memory, typically using a columnar and compressed format. Some solutions are even able to perform work on the data without first decompressing it.
- Fast. The dream of BI — particularly for interactive analysis tools — has always been “speed of thought” analysis. Thanks to the above point and thanks to additional performance optimizations (e.g., to expoit CPU cache locality), this dream is becoming a reality.
- Directly connected. Next-generation BI tools generally connect directly to the underlying source databases (and/or the Internet) to capture data. This means they must also have basic data integration capabilities both so they properly align data from different systems and dynamically refresh it.
- Schema-free. In order to accomodate semi-structured information and to be able integrate information from different sytems, next-generation BI does not require the up-front definition of a schema. Instead, relationships among data (e.g., hierarchy) are discovered dynamically.
- Beautiful. While this is best exemplified by Tableau (where visualization is the principal focus) next-generation BI tools generally provide beautiful visualizations that are more powerful than the basic report and bar chart. (Note that I named a name here because I consider Tableau mid-stage, not early-stage.)
- Mobile. Next-generation BI tools typically assume a brower-based client and often the need to create device-specific clients (e.g., a native iPad app) to supplement it. Some companies focus exclusively on mobile BI.
- Neutral. Next-generation BI tools exploit the fact that a multi-billion dollar vacuum was created in the market when the BI leaders were consolidated and became units of IBM (e.g., Cognos) or SAP (e.g., BusinessObjects).
In many ways, next-generation BI takes us full circle back to the days of Cognos PowerPlay and its desktop-resident PowerCube (i.e., hypercube) — except that the cube is now virtual, schema-free, of effectively unlimited size, and contains no precalcuated aggregates. But like that era, the cube in many ways obviates the data warehouse infrastructure underneath it. After all, if you can fit your entire data set in memory and dynamically calculate the answer to any question at high speed, then why do you need a data warehouse full of precalculated aggregates again?
The answer is “you do” for many cases (e.g., history, data cleansing) — but certainly not for all of them. I thus see a “middle squeeze” on the data warehouse market in the future.
- For most applications of normal size and analytic complexity, people will use next-generation BI on top of raw data sources, unless they have very messy data or a need for extensive history.
- For large applications (i.e., big data) and/or high analytic complexity, people will use advanced analytic platforms (e.g, Aster Data). This, of course, begs the question whether anyone is working on BI tools that exploit and optimize the new, high-end analytic engines and the answer to that question is happily “yes” as well.
I really enjoyed this post, Dave – a clear and succinct snapshot of many of the table stakes traits of next generation BI. One item I would add to the list would be the ability to take advantage of unstructured data, encompassing capabilities like search, data integration from unstructured sources, and text analytics, to name a few. I see this requirement coming from the market as analytic applications move to encompass unstructured enterprise data as well as public data like forums and social content (e.g., there’s a dip in the sales of this product – what are my customers saying about it?). And in keeping with the theme of your post, I see this as a problem that a number of next generation BI vendors are starting to take on, although I have my personal bias about who’s the farthest ahead :)
Thanks Adam. Given my unstructured data background at MarkLogic I was perhaps overcompensating, down-playing unstructured data in the post, basically putting it beneath “schema free.”
So to be clear, I basically agree with your point — accomodating and incorporating unstructured and semi-structured information — possibly sourced not from internal systems but from the Interent at large — is indeed one of traits of next-generation BI, even if not articulated as a separate bullet above.
Great blog post. I think you are 100% on with your analysis, the only point I would add is BI will become more driven by the business users and be not only self-service but collaborative. That’s they focus we have been taking with Yurbi. The key is to ensure the security is in place so that IT groups and Security teams can feel safe to empower their end users with BI capabilities.
I found your blog from the BI Daily blog, I’ll add it to my list of blogs to watch. Thanks
Dave, great post. One friend used to say, BI is summarized of data on a set of metadata and get it delivered effectively. Your summary covered a lot of changes in each aspect – fast, in memory calc on summarization, schema free, source schema driven, and lastly delivery (beautiful, mobile, browser). I notice one of the hot trends missed from your post. Cloud. If more apps are on cloud with their schema published as API objects and can be pulled across internet, and the BI system itself has started to overcome some of the schema integration, aggregation, and delivery issues, would it facilitate a cloud based BI system? Although in reality, I haven’t heard of successful start-ups (to date) in this area… Any comment on why?
Several folks have noted that I left cloud off the list, I suppose on purpose because I view cloud as a delivery mechanism and not intrinsic to the product. I think reaching into the cloud and using APIs to get data is critical. Whether BI happens in the cloud or not, I don’t know. I’ve always believed that BI alongside apps (e.g., the Salesforce reporting module) makes perfect sense in the cloud and I think Good Data is doing a well, good, job of trying to become the Crystal Reports of that space.
But pure cloud BI, where the fact that the tool runs in the cloud, is multi-tenant, etc., has had a checkered history (e.g., think LucidEra). In some sense I’d argue that pure SaaS BI is generation N-1 and the current generation of startups that I’ve seen are not focused on delivery model, per so. They view cloud as additional product requirement[s] and not as the core/differentiating focus.
Pingback: Highlights from the 2011 Wisdom of Crowds Business Intelligence Market Study | Kellblog
Pingback: SOFTWARE & JAZZ » Blog Archive » The case for Editorial BI