Posted on

Data Analytics

The term ‘Data Analytics’ is becoming more and more commonly used…

Data analytics are also the end game of the BI System Builders’ Cornerstone Solutions®.  But what are data analytics? This article describes the term as used by the BI System Builders. While experts in other organisations may have a different definition this paper will serve to make the BI System Builders’ thinking clear. We will start by considering the  ‘classic’ data analytics first becoming popularised pre-millenium and then move on to the new breed data analytics made possible through new technologies.

Firstly, it’s important to note that the term data analytics is frequently used interchangeably with other names such as analytical applications, business analytics, and as a component of Performance Management Applications to name but a few.  BI System Builders only use one terminology, that of ‘data analytics’. That said, data analytics tend to fall into three categories, business analytics, statistical analytics, and predictive analytics. In practice these analytic types may be combined.

Now, all organisations capture data. This may be through sales invoices, captured in a call centre, via a website, delivery notes from suppliers, research questionnaires, financial transactions, point of sale transactions or in the area of scientific research or engineering and so on and so forth. The data captured is in a raw format and will usually be disjointed across several of your data capture and transaction systems. This data can be very difficult to interpret so an activity may be undertaken to bring the data into some form of structured database tables even if it’s just a one to one mapping. You may hear people referring to these database tables as ‘landing tables’, ‘staging tables’, or ‘Operational Data Store (ODS) tables’. ODS tables may be ‘raw’ in form or may be highly structured though a data modelling technique known as entity modelling or third normal form (3NF or BCNF) modelling. Either way all these table types can be very complex to ask questions against. At this stage data viewed in reports is often referred to as operational reporting.  The data in operational reports is not summary but low level, detailed, granular, disjointed, and may include codes that don’t seem to make any sense. Have you ever tried to read through and make sense of a million rows of limited format technical data and codes in a day? That detail of data may have its use for certain operational reporting purposes but for business user purposes it needs to be more readable and needs to be presented at a higher level being consolidated or summarised in some way to make it readable.

The transaction/source system/raw data captured has been increasing in high volumes over the years. This raw data may be structured or unstructured data and can also be highly volatile in nature. Furthermore the exponential growth of the internet, mobile devices, social media, scientific research, and technology has meant the potential for enormous amounts of detailed personal information to be captured everyday – your web clicks, your buying behaviour, your communications, your location, GPS co-ordinates, your posts, your reviews, etc., etc., hence the entrance of GDPR in 2018.

Historically, business intelligence tools have been evolved to extract, clean and format the raw data then join it up together turning it into valuable and useful business information presented through data analytics.  The business intelligence tools are relevant and highly valuable when used in conjunction with a structured data warehouse. The data is often summarised and may also be pre-calculated in other ways as well. The information can then be used to discover insights into your business or organisation. The reports are sometimes shared across the organisational enterprise and can be based on enterprise wide data in an enterprise data warehouse (EDW), hence the term, enterprise reporting.

Let’s take a look at the classic data analytic capability in the data warehouse and then consider data analytics in the new Big Data era. Data analytics go further than simple reporting by adding extra insights/intelligence into the reports. This is done in several ways.  Some of the classic ways are the use of dynamic parameters for on the fly analysis such as period on period,  ‘drill down drill up’ through hierarchical data structures, a drill across capability linking business process areas across the business, predictive engines, and ‘what if’ analysis that allow the user to play out different scenarios by changing the values of variables in the data analytic.

Dynamic parameters allow the user to refresh the data in the data analytic and change the question being asked by selecting contextual values in a prompt. For example with period on period analysis you may start by viewing sales today compared with sales yesterday.  The user can then easily change the data analytic to compare current month to date sales versus last month to date sales or last year’s month to date sales etc. The simple data analytic below is probably cleverer that it looks as it includes several calculated fields and all the analysis periods can be changed on the fly. When they are changed the other dependent fields automatically recalculate. The data analytic also has a hierarchical data structure drill down capability and dynamically changing sentences which automatically capture values such as the name of the city with the highest percentage change.

Business Analytics

Building on the concept of drill down and drill up, this is a technique used to navigate through a hierarchical structure in your data. For example you may capture sales data by location. In an data analytic with a hierarchical data structure you may view sales for all locations and then with a click of the mouse ‘drill down’ to sales by region, and then drill down again to sales by city and down to individual sales outlets. This can all be achieved within a single data analytic.

The drill across capability allows you to navigate through a series of linked data analytics. These are often based on a logical work process flow.  The linkage is achieved through code written behind the scenes and invisible to the user. Data analytics can be clever enough to know the context of the information you are viewing such as location, time period and product and pass this context to the next data analytic.

Predictive and ‘what if’ capabilities allow the user to play out different scenarios. For example you could measure the impact of inflation. To do this a percentage value would be input into a dynamic parameter prompt.  An algorithm in the data analytic would then read the value and recalculate itself to show you the impact. Other algorithms may be designed that are complex in nature and may use statistical analysis techniques such as regression and correlation.

There are many other things that you can achieve with data analytics such as cycle time analysis.  The picture below is a cycle time chart. In this case the chart is analysing the customer order actual cycle time but the technique can be applied to any business process cycle time. The information has great value. If you can measure the duration of individual stages within a business process you can identify those that are least efficient. By then addressing any inefficiency within a stage you can make it more efficient, reducing its duration and consequently cost, thus improving the profitability of the process.

It is also common to measure things such as Key Performance Indicators (KPIs), customer churn, customer life time value, stock turn. It is possible to develop time series analysis, strategy maps, balanced scorecards, and six sigma based statistical process charts across business process areas, not just one process area in isolation. Data analytics based on a business intelligence system could technically be used to combine HR data with finance data, and finance data with supply chain data and so on and so forth. The bottom line is that data analytics will help you find efficiency and effectiveness in your business according to your needs. This type of data analytic is already used by many organisations although others still struggle to realise them. Other examples of data analytics are also accessible through social media platforms  including Google Analytics, LinkedIn Analytics, and YouTube Analytics.

Business Analytic Cycle Time

However, we no longer live in the  slowly changing world of the data warehouse alone. The cost of technology is reducing, the 64 bit processor is here and large amounts of RAM can now be exploited. Furthermore, technological advancements have yielded in-memory applications such as SAP HANA and IBM’s DB2 Blu Acceleration and distributed architectures and file systems such as Apache Hadoop. The combination of these things means that enormous volumes of data can be ‘released’ from the systems in which they have been captured and processed fast – near real-time, and real-time fast. Things that were once beyond budget are now starting to come within budget.

Earlier I asked the question about have you ever tried to read through a million rows of data? I wasn’t joking, I’ve witnessed business users attempting to do this is in a report and then find the bits of information that they needed. Of course, they give up quite quickly and don’t get repeat opportunities because of the system degradation this type of  report processing has historically caused. Now I’ll ask another question, “Have you ever tried to read though a hundred billion rows of data in a report?” Preposterous? – yes, impossible to process? – no, some organisations now capture data at the terabyte plus scale.

It is impossible to make meaning out of such high volume, highly volatile and disparate data as is now becoming available without new breed data analytics. But these data analytics do not need to be reactive, running against previously processed data, they are now being exploited in a pre-emptive way. Consequently the term ‘data scientist’ has been popularised and is frequently closely associated with  data analytics. Volumes of raw data may be so huge that they are referred to as a data lake. On these huge data lake volumes an algorithm or logic may be executed on the unstructured data in the form of programs before any other processing apart from data capture has occurred. Further programs may also include ‘learning models’ and seek to find previously hidden relations within the data.  Historically, a related albeit more simple type of activity was referred to as data mining, now the activity may be associated with the names, Artificial Intelligence (AI), Machine Learning, and data science. Ultimately, they are a ‘pre-emptive strike’ data analytic searching data values and outputting a result set of some type. As sophisticated as the new data scientists may be with algorithms, coding, and complex modelling the concepts of using decision engines, learning models, predictive engines, data mining, Business Activity Modelling (BAM), trending, time series analysis, correlation, regression, tests of significance, pattern detection,probabilities, disparate unstructured source mapping, and the identification homogeneous behaviours is not new. However, now the technology and the data is available these things will be exploited in data analytics for business and research like never before. For a clear example of this consider the idea ‘big data means marketing science’, but the topic of data collection, technological capabilities, privacy, and ethics deserves an article all of its own…

Posted on

Death Of The Cube – Long Live The Cube!

OLAP Cubes

OLAP Cubes

The acquisition of BusinessObjects by SAP paved the way for a very welcome tighter integration between the two softwares. One of the challenges coming out of that tighter integration was the performance of Web Intelligence against an OLAP universe generated on SAP cubes and BEx Queries. The reality of SAP project implementations was that SAP Netweaver experts designed large cubes and large queries. And why not; after all this was the OLAP world?! Large SAP cubes and large BEx Queries make sense for OLAP.

However, Web Intelligence is not an OLAP tool, it builds a cache of data referred to as a ‘microcube’. Note the word ‘microcube’. Attempting to pass large volumes of data from an OLAP query to the microcube could cause the Web Intelligence engine to perform poorly or crash. BISB have observed this on numerous occasions when undergoing performance testing at client site. Problems with the version of Explorer dependent up on the Web Intelligence engine have also been observed for the same reason.

But failing to process large volumes of data was not a weakness of Web Intelligence. On the contrary, Web Intelligence was designed for smaller, fast, ad hoc queries. Users experiencing problems with large volumes of data and Web Intelligence could consider the use of Crystal Reports. Crystal Reports uses a different cache infrastructure to Web Intelligence.

The above mentioned data volume issues have made the SAP BI 4.0 road map very welcome. Using the new Data Federator connectivity through the SAP BusinessObjects BI 4.0 universe means that the SAP MDX engine (OLAP) is bypassed. This removes one of the big issues of the SAP OLAP data volumes, namely MDX crossjoins. Other development means that the BI 4.0 universe now has connectivity to SAP HANA. If you have the budget available this makes SAP HANA highly desirable for Big Data and Analytics.

Finally, ardent OLAP users that cannot live without a cube have not been left out in the cold. BI 4.0 ushered in the end of the Voyager OLAP tool, replacing it with the new Advanced Analysis for OLAP tool.

The view expressed in this article is from BISB and not necessarily SAP. Russell Beech was Senior Analyst in the BusinessObjects Analytic Applications Division for almost six years. Check out Web Intelligence In Under Three Minutes here.

Posted on

Agile Data Warehouse Design: Collaborative Dimensional Modeling, from Whiteboard to Star Schema by Lawrence Corr and Jim Stagnitto

BEAM Business Event Analysis & Modeling

BACKGROUND

I first worked with Lawrence Corr back in 2002 whilst I was designing content in the Analytical Applications Division (AAD) of the BusinessObjects Product Group.  At thatBEAM Business Event Analysis & Modeling time Lawrence was engaged as an external consultant to BusinessObjects, critiquing and advising us (AAD), on our BusinessObjects data warehouse design. Back then Lawrence also gave me my first formal dimensional modeling training when I attended his Data Warehousing Design Techniques course. Lawrence already had a very impressive reputation and was closely associated with Ralph Kimball.

It was therefore of interest to me when Lawrence said that he was writing a new book entitled Agile Data Warehouse Design.  Here’s what I found…

 

AGILE

There are two initial points to make about the book. Firstly, about the title of the book Agile Data Warehouse Design. Despite what the title might initially suggest I did not find the book to be about delivering a data warehouse using existing Agile techniques such as Scrum and extreme Programming (XP). It is rather about a structured method of bringing together Business Intelligence requirements analysis and dimensional modeling techniques using an Agile mindset.  The goal being to deliver logical models that work, in a highly time efficient fashion.  As such the Agile Manifesto is listed at the rear of the book and it is easy to see how the methods described meet the aims laid out in the manifesto.

 

PRACTICAL METHOD

Secondly, Agile Data Warehouse Design is a pragmatic book. It is not just agile theory alone. It will provide you with practical techniques, artifacts, and tools that will enable you to model successfully. I say that because I have already implemented these techniques, known as BEAM, extensively at a leading insurance company, at a leading car manufacturer working across all their vehicle brands and for a well known high street retailer. I have found that business users became actively engaged when introduced to the BEAM technique of the 7Ws (more on the 7Ws later).  Furthermore the BEAM tools made it easy for end users to contribute in an intelligent and structured way. That said, the business users did not need to understand the BEAM techniques themselves; in fact I never mentioned that we were using BEAM at all and they didn’t need to know. They simply attended the interviews and enjoyed having their brains picked and taking joint ownership of the developing dimensional model.

 

BI BREAKPOINTS

I became all the more interested in reading Agile Data Warehouse Design when I began to realise that it tackles head on several key ‘BI Breakpoints’:  the term used by BI System Builders to describe weaknesses in an End to End BI solution that can become points of failure. While the term is not explicitly used in the book it quickly became clear to me that the BEAM method will help developers address the specific BI Breakpoints between Business Analysis and Data Warehouse design.  As such, I found Agile Data Warehouse Design to be highly complimentary to the Cornerstone Solution® BI method.  The Cornerstone Solution® End to End BI method is used by BI System Builders to address BI Breakpoints. You can read more about BI Breakpoints here.

 

WHICH BI BREAKPOINTS DOES BEAM ADDRESS?

BEAM addresses BI Breakpoints around business analysis and dimensional model design.  A key issue for effective dimensional modeling, that I’ve faced many times, is that it requires the combination of three different contributing skill sets: Business Domain Expertise, Business Analysis, and Dimensional Modeling. The domain expertise is provided by the business. However, it is the role of the business analyst (BA) to extract that expertise, understand the business process area and then document the business requirements. To do this successfully requires the ability to ask the right questions.  Once the BA’s document is available it is translated into a dimensional model by the Business Intelligence and Data Warehouse (BI/DW) team.

Generally speaking I observe that BAs will have a predominantly business background while  dimensional modelers (DMs) a technical one.  Frequently a BA is assigned to go to the business and gather user requirements, the result of which is a copious document. Once the document is complete it is handed over to the BI/DW team to work with. Although the document is useful, typically it will not explicitly describe critical dimensional modeling design elements such as fact granularity, and fact and dimension table types and relationships as required by the BI/DW team for development purposes. Consequently, this handover can become a BI Breakpoint.

The BI/DW team will attempt to interpret the business analysis document as best they can. However, issues can arise because the BI/DW team had no involvement during the analysis stage and could not ask pertinent questions whilst the business analysis was being undertaken.  After sign-off of the business analysis document the BA may move onto another project and not be available to provide further help. As contractors and consultants are often used as BA’s they may even have left the business all together. This can cause a chasm of understanding to open up between the BI/DW team and what the business users had been describing and requesting.  Needing clarification or finding information missing and not knowing or wanting to approach the business again the BI/DW team may fall back onto something that is more securely under their control as a means to drive their modeling effort – source system data analysis.

The risk of building out dimensional models based on source system analysis is that the final tables will be close in design to the source data but may not model the business process area or meet business user needs.  The tables may not meet business requirements and they may not be a true dimensional design at all. To my mind this is a failing, because ignoring for the moment the new SAP HANA, I have always found dimensional models to be the most effective performance design for use with SAP BusinessObjects tools against a relational database and the best way to think of business process measurement in general.

To help avoid the BI Breakpoint that can occur between the BA and the BI/DW developers we have the notion of cross-functional teams.  A cross-functional team is superior to the structure previously mentioned.  The team members work closely and simultaneously together often in the same project room. The DM from the BI/DW team sits in on the BA’s interviews with the business users and starts to construct the logical model design. The dimensional modeler can ask clarification questions directly to the BA and business user at any point in the process. Furthermore the evolving logical model design can be frequently replayed to the rest of the team to confirm it. In my experience the cross-functional team has been more successful than the polarised BA and BI/DW (chasm-forming) teams. BEAM takes the concept of the cross-functional team much further and provides an intelligent and effective framework for the BA and BI/DW teams to work together in.  Following the BEAM method is an effective antidote to creating BI Breakpoints.

 

WHAT IS THE BEAM METHOD?

BEAM stands for Business Event Analysis and Modeling.  As the name suggests it combines elements of requirements analysis and data modeling. Its key concept is to use 7 dimensional types (the 7Ws) to identify and then elaborate business events. BEAM concentrates on business events rather than known reporting requirements so as to model whole business process areas.  This provides a major advantage.  Modeling a business process area yields a design that can be readily scaled as requirements grow. Modeling for a set of reporting requirements alone can lead to a narrow solution. ‘Narrow’ because the design may not lend itself to be scaled when new requirements are on boarded. Therefore, the BEAM approach helps avoid the BI Breakpoint of non-scalability. BEAM’s 7W approach also lays a solid foundation for ad-hoc reporting and self-service BI by teaching business users – by stealth – to think dimensionally.

The 7Ws used by BEAM are: Who, What, When, Where, How, How Many, and Why.  A similar conceptual technique is used in investigative journalism to ensure full story reporting coverage. For a specified business process area the BEAM idea is to identify event stories by asking a ’who does what’ question and then expressing the answer as a simple story.  An example of this would be ‘traders buy commodities’.  A series of these ’who does what’ stories are captured and then the remainder of the 7Ws such as the ‘when’ and ‘where’ are asked to drive out their interesting details. All the results are documented in a BEAM table template.

The BEAM table template is one of several tools employed, you will also learn how to use the BEAM tools of hierarchy charts, timelines, event matrices, and enhanced star schemas. The BEAM method will then take you through modeling events, dimensions, processes and star schemas to provide working software and documentation as detailed in the Agile Manifesto.

From the information gathered in the Business Event Analysis stage it is then possible to easily identify dimension and fact table types. Dimension and fact table patterns are explained in the second half of the book  ‘Modeling’.  If you are new to dimensional modeling you will learn much from the vast design and implementation experience of Lawrence and his co-writer Jim Stagnitto. The BEAM method and notation walks you through a natural continuum from the interview stage right through to the end dimensional model.

When the BEAM method is properly understood and implemented it will effectively bridge the gap (BI Breakpoint) between the BA and the DM. Both the BA and the DM can work together using BEAM or for someone with hybrid skills the two roles can become one. In summary Agile Data Warehouse Design is a thoroughly well written book that addresses BI Breakpoints and brings with it four key benefits.  It will show you how to practically apply an effective combined analysis and modeling method (BEAM). It will help engage business communities so that full business process areas can be modelled making your solution scalable. It will lower costs to the business by reducing analysis and modelling time. It will reduce the risk of a project struggling by delivering working software and documentation on time.

You can buy Agile Data Warehouse Design from Amazon and find out more about BEAM and matching agile/dimensional modelling courses on Lawrence’s Decision One Consulting web site.

Posted on

BusinessObjects Web Intelligence in Under 3 Minutes

Web Intelligence in Under 3 Minutes

So what is Web Intelligence and what can it do? We’ve created a short video of a live Web Intelligence demo. The video covers creating a new query, a table, a pie chart, a cross-tabulation, a calculation within a variable and an alert, drilling on a chart and then down through a product hierarchy in a table, finishing by creating a section.

Posted on

BI System Builders Display at Chamber of Commerce Event- May 2018

BI System Builders attended an event organised by the Coventry & Warwickshire Chamber of Commerce on 4th May, Star Wars Day. It was a great opportunity to demonstrate our capabilities around business intelligence, data analytics, and data warehousing. Many local businesses attended and valuable conversations were had. Although some of the attendees taking an interest in artificial intelligence and machine learning seemed to be out of this world!

Posted on

BI System Builders at the Houses of Parliament

On the 4th of December 2017 BI System Builders founder, Russell Beech attended the South Warwickshire at Parliament event held in the Members’ Dining Room at the Houses of Parliament. The event was organised by Jonathan Smith and hosted by Nadhim Zahawi, MP for Stratford-upon-Avon.

The invitation to BI System Builders was made by Professor Simon Swain, Pro-Vice-Chancellor at the University of Warwick (pictured left).  The event brought together business and academic leaders and senior government officials. There were five short speeches made including one by Professor Swain and another by Andy Palmer, CEO at Aston Martin. Russell commented that he was looking forward to BI System Builders continuing to build relationships with the University of Warwick.  Read the article from the University of Warwick here.