In my daily trail through various technology sites and blogs, I came across an article written by Timo Elliot, “The Last Decade of BI Best-Practice Architecture is Rapidly Becoming Obsolete.” It gives a great insight into the changing face of Business Intelligence and the technological improvements made over the years, to where we are approaching right now- where the combination of BI technologies mean that analytical platforms are becoming increasingly efficient, cost-effective and faster thanks to a number of developments including in-memory analysis, column databases, customer appliances, etc with the combination of operation and analytics in their vision.
Particularly interesting is the first chapter of the article which outlines the principles behind one of the first Business Intelligence solutions, “Lyon’s Electronic office” (1951). Lyons was the Costa coffee of its era with large tea rooms open 24 hours a day, where hundreds of cups of tea were consumed on a daily basis. Lyon’s tea houses like businesses of today faced numerous business challenges as a result of the sheer scale of their organisational activities; such as ordering correct stock, reducing surplus ordering, payroll, etc. So the LEO was created in an attempt to alleviate these issues- it optimised profitability and minimised waste via calculating how many buns and tea cakes should be produced for the next day based on the latest purchase data available- pretty impressive for its day. The LEO is extremely significant in terms of the evolution of BI into how it is approached and applied within the analytic solutions of today. As Elliot describes it, “The very first commercial computer was already about Business Analytics”. However, the operation process was slow and expensive via the use of thousands of Vacuum tubes which used mass amounts of heat and energy. As a result, the LEO was soon to be replaced by a more efficient, effective technology, the transistor- invented in 1947. The transistor; cheaper, more reliable, faster and smaller, revolutionised computer technology- enabling computers to do more than ever, this ever more so with the development of transistors over time, whereby transistors were “miniaturized” and packed in their millions onto single chips enabling “unthinkable possibilities”
After a little History lesson in BI, Elliot draws upon an interesting analogy, arguing that as BI and Data warehousing practices of today are that of the “vacuum tube era” and we are now slowly approaching the “transistor era”- more efficient, faster analytic solutions able to cope with huge amounts of data- perhaps a little loose as far as analogy’s go, but still he gets his point across and certainly makes up for it with his arguments for this. Elliot argues that the last decade of Business Intelligence Best Practice, like the LEO Vacuum technology, has been slow, expensive and painful and the traditional BI processes were holding back our success in the plight for fast, easy, cheap and reliable Business Analytics and reporting. So what does BI best practice of the last decade look like? Elliot suggests that it begins with the selection of business applications that gather the data we would like to analyse. However, systems are slowed down if we do too much reporting, so a copy is created- known to the technology savvy of the BI world as an ODS (Operational Data Store). In addition, Data Marts and Data Warehouses are created using ETL technology to alleviate incomplete and incompatible data as a result of the ODS being incapable of storing history. Businesses want to store lots of information, yet quick delivery times are impossible to achieve without complex database structures and indexes- subsequently adding to the size and complexity of the data warehouse. The result? – A solution that works well enough, will get the job done, but is slow, costly and complex (very similar to the LEO Vacuum system).
So faced with these challenges, what has been done to alleviate these issues and how far have we come in our journey to more effective Business Analysis? Elliot goes into some depth about the use of new technology to improve analytical processes- the most significant in his eyes being the development of “In- memory processing” and the reduction in cost of memory storage over the last decade, enabling data to now be stored on memory as opposed to the traditional method of storage data on disk- whereby the retrieval of data is slow. Though, I would argue that vendors have attempted to alleviate this prior, with Memory resident analysis which allows speedier access to data on key functions within an organisation, which goes a long way to solving the problem without the purchase of a full scale in-memory solution.
However, much is to be said of Elliot’s argument that in- memory process will revolutionise Best BI practice: Essentially, via use of in-memory process we could eliminate entirely the use of data marts and database optimisation tools (previously used by vendors to increase the efficiency of disk storage) such as aggregates and indexes, thus simplifying the data retrieval process. Elliot also emphasises in some detail the significance in the combining of in-memory processing and column data stores (his analogy of his and his wife’s filing systems is a particularly quirky way to explain the benefits of column data storage). Thanks to the ability to store similar types of data together in groups, information is stored more effectively and can be compressed, previously not possible in row based storage techniques. Hence, you can store more data in the same physical space, shrinking the data warehouse back down to a size similar to the raw data used to create it . This in turn reduces the amount of memory you need to scan and increases processing speed.
Column based data storage solutions do have their downsides-when using disk storage, column based data (due to its size and capacity) has an extremely lengthy load time, which, despite the clear benefits of column based storage has, until now, limited its appeal in the market place. Yet, Elliot raises an interesting point, if column based storage was combined with in-memory processing, due its compact nature we can store an entire data warehouse in-memory. And thanks to the speed of in-memory processing loading times are no longer a problem- The first step, argues Elliot, to the new generation of Business Analytics.
Also needing to be brought into the mix for the New Generation of analytics and certainly worth a mention also are hardware acceleration appliances and in database calculations (via an analytics calculation tool such as ‘PeopleSoft’). Elliot argues that the combination of the four technologies will ultimately lead to unrivalled analytics process which achieves the same goals as traditional BI architecture, but does it faster and more effectively and could potentially lead to further developments if this was built upon further, such as the ability to use the same platform for both transactional and analytical processing if ACID compliance and dual capability for row and column data storage was also enabled.
Despite me seemingly being an evangelist for Elliot’s new vision for the development and improvement of business analytics I do see some significant flaws in his ideas. Elliot is pro Hardware acceleration tools, however I think it will take a significant amount of time before Hardware acceleration tools gain acceptance in the marketplace, due to the complexity in programming and their unsuitability for certain organisations, for example the financial sector, as they have to bolt into proprietary hardware. He also lacks emphasis on the importance of the Cloud in the future of analytics and Business Intelligence which I feel deserved more of a mention, if his proposed process of combining the four mentioned technologies into a “super” analytics process was then placed in the cloud, this would allow for more mobile Business intelligence and ensure that data collaboration was made easier- so faster, more efficient technology with the ability to take analytics and reporting organisation-wide and even externally. Elliot also fails to the see the switch from the importance of the delivery of information to the analysis of that information. So yes, the efficiency, effectiveness and speed in which information can be delivered is paramount, however there seems to be a switch in importance of late to the access and ability to analyse, understand and make decisions up data; this is elucidated in a recent Gartner report,” The market is shifting from storage and access to delivery and comprehension.” Perhaps more emphasis could have been placed on this within Elliot’s article.
You can read Elliot’s article via visiting the following page. Some other interesting articles on Business Intelligence and Business Analytics also which are definately worth a look. http://smartdatacollective.com/timoelliott/33492/why-last-decade-bi-best-practice-architecture-rapidly-becoming-obsolete
