Posted on

Successful Business Intelligence – How to DIY Your Own Centre of Excellence

Business Intelligence strategy is key to success.

A Centre of Excellence is an effective way to ensure your Business Intelligence Programs and Projects are successful. In this video Russell Beech gives you advice and tips on how to DIY your own Centre of Excellence. Each organisation is different but the ideas presented will be valuable to you in creating your own CoE.

Posted on

Data Analytics

The term ‘Data Analytics’ is becoming more and more commonly used…

Data analytics are also the end game of the BI System Builders’ Cornerstone Solutions®.  But what are data analytics? This article describes the term as used by the BI System Builders. While experts in other organisations may have a different definition this paper will serve to make the BI System Builders’ thinking clear. We will start by considering the  ‘classic’ data analytics first becoming popularised pre-millenium and then move on to the new breed data analytics made possible through new technologies.

Firstly, it’s important to note that the term data analytics is frequently used interchangeably with other names such as analytical applications, business analytics, and as a component of Performance Management Applications to name but a few.  BI System Builders only use one terminology, that of ‘data analytics’. That said, data analytics tend to fall into three categories, business analytics, statistical analytics, and predictive analytics. In practice these analytic types may be combined.

Now, all organisations capture data. This may be through sales invoices, captured in a call centre, via a website, delivery notes from suppliers, research questionnaires, financial transactions, point of sale transactions or in the area of scientific research or engineering and so on and so forth. The data captured is in a raw format and will usually be disjointed across several of your data capture and transaction systems. This data can be very difficult to interpret so an activity may be undertaken to bring the data into some form of structured database tables even if it’s just a one to one mapping. You may hear people referring to these database tables as ‘landing tables’, ‘staging tables’, or ‘Operational Data Store (ODS) tables’. ODS tables may be ‘raw’ in form or may be highly structured though a data modelling technique known as entity modelling or third normal form (3NF or BCNF) modelling. Either way all these table types can be very complex to ask questions against. At this stage data viewed in reports is often referred to as operational reporting.  The data in operational reports is not summary but low level, detailed, granular, disjointed, and may include codes that don’t seem to make any sense. Have you ever tried to read through and make sense of a million rows of limited format technical data and codes in a day? That detail of data may have its use for certain operational reporting purposes but for business user purposes it needs to be more readable and needs to be presented at a higher level being consolidated or summarised in some way to make it readable.

The transaction/source system/raw data captured has been increasing in high volumes over the years. This raw data may be structured or unstructured data and can also be highly volatile in nature. Furthermore the exponential growth of the internet, mobile devices, social media, scientific research, and technology has meant the potential for enormous amounts of detailed personal information to be captured everyday – your web clicks, your buying behaviour, your communications, your location, GPS co-ordinates, your posts, your reviews, etc., etc., hence the entrance of GDPR in 2018.

Historically, business intelligence tools have been evolved to extract, clean and format the raw data then join it up together turning it into valuable and useful business information presented through data analytics.  The business intelligence tools are relevant and highly valuable when used in conjunction with a structured data warehouse. The data is often summarised and may also be pre-calculated in other ways as well. The information can then be used to discover insights into your business or organisation. The reports are sometimes shared across the organisational enterprise and can be based on enterprise wide data in an enterprise data warehouse (EDW), hence the term, enterprise reporting.

Let’s take a look at the classic data analytic capability in the data warehouse and then consider data analytics in the new Big Data era. Data analytics go further than simple reporting by adding extra insights/intelligence into the reports. This is done in several ways.  Some of the classic ways are the use of dynamic parameters for on the fly analysis such as period on period,  ‘drill down drill up’ through hierarchical data structures, a drill across capability linking business process areas across the business, predictive engines, and ‘what if’ analysis that allow the user to play out different scenarios by changing the values of variables in the data analytic.

Dynamic parameters allow the user to refresh the data in the data analytic and change the question being asked by selecting contextual values in a prompt. For example with period on period analysis you may start by viewing sales today compared with sales yesterday.  The user can then easily change the data analytic to compare current month to date sales versus last month to date sales or last year’s month to date sales etc. The simple data analytic below is probably cleverer that it looks as it includes several calculated fields and all the analysis periods can be changed on the fly. When they are changed the other dependent fields automatically recalculate. The data analytic also has a hierarchical data structure drill down capability and dynamically changing sentences which automatically capture values such as the name of the city with the highest percentage change.

Business Analytics

Building on the concept of drill down and drill up, this is a technique used to navigate through a hierarchical structure in your data. For example you may capture sales data by location. In an data analytic with a hierarchical data structure you may view sales for all locations and then with a click of the mouse ‘drill down’ to sales by region, and then drill down again to sales by city and down to individual sales outlets. This can all be achieved within a single data analytic.

The drill across capability allows you to navigate through a series of linked data analytics. These are often based on a logical work process flow.  The linkage is achieved through code written behind the scenes and invisible to the user. Data analytics can be clever enough to know the context of the information you are viewing such as location, time period and product and pass this context to the next data analytic.

Predictive and ‘what if’ capabilities allow the user to play out different scenarios. For example you could measure the impact of inflation. To do this a percentage value would be input into a dynamic parameter prompt.  An algorithm in the data analytic would then read the value and recalculate itself to show you the impact. Other algorithms may be designed that are complex in nature and may use statistical analysis techniques such as regression and correlation.

There are many other things that you can achieve with data analytics such as cycle time analysis.  The picture below is a cycle time chart. In this case the chart is analysing the customer order actual cycle time but the technique can be applied to any business process cycle time. The information has great value. If you can measure the duration of individual stages within a business process you can identify those that are least efficient. By then addressing any inefficiency within a stage you can make it more efficient, reducing its duration and consequently cost, thus improving the profitability of the process.

It is also common to measure things such as Key Performance Indicators (KPIs), customer churn, customer life time value, stock turn. It is possible to develop time series analysis, strategy maps, balanced scorecards, and six sigma based statistical process charts across business process areas, not just one process area in isolation. Data analytics based on a business intelligence system could technically be used to combine HR data with finance data, and finance data with supply chain data and so on and so forth. The bottom line is that data analytics will help you find efficiency and effectiveness in your business according to your needs. This type of data analytic is already used by many organisations although others still struggle to realise them. Other examples of data analytics are also accessible through social media platforms  including Google Analytics, LinkedIn Analytics, and YouTube Analytics.

Business Analytic Cycle Time

However, we no longer live in the  slowly changing world of the data warehouse alone. The cost of technology is reducing, the 64 bit processor is here and large amounts of RAM can now be exploited. Furthermore, technological advancements have yielded in-memory applications such as SAP HANA and IBM’s DB2 Blu Acceleration and distributed architectures and file systems such as Apache Hadoop. The combination of these things means that enormous volumes of data can be ‘released’ from the systems in which they have been captured and processed fast – near real-time, and real-time fast. Things that were once beyond budget are now starting to come within budget.

Earlier I asked the question about have you ever tried to read through a million rows of data? I wasn’t joking, I’ve witnessed business users attempting to do this is in a report and then find the bits of information that they needed. Of course, they give up quite quickly and don’t get repeat opportunities because of the system degradation this type of  report processing has historically caused. Now I’ll ask another question, “Have you ever tried to read though a hundred billion rows of data in a report?” Preposterous? – yes, impossible to process? – no, some organisations now capture data at the terabyte plus scale.

It is impossible to make meaning out of such high volume, highly volatile and disparate data as is now becoming available without new breed data analytics. But these data analytics do not need to be reactive, running against previously processed data, they are now being exploited in a pre-emptive way. Consequently the term ‘data scientist’ has been popularised and is frequently closely associated with  data analytics. Volumes of raw data may be so huge that they are referred to as a data lake. On these huge data lake volumes an algorithm or logic may be executed on the unstructured data in the form of programs before any other processing apart from data capture has occurred. Further programs may also include ‘learning models’ and seek to find previously hidden relations within the data.  Historically, a related albeit more simple type of activity was referred to as data mining, now the activity may be associated with the names, Artificial Intelligence (AI), Machine Learning, and data science. Ultimately, they are a ‘pre-emptive strike’ data analytic searching data values and outputting a result set of some type. As sophisticated as the new data scientists may be with algorithms, coding, and complex modelling the concepts of using decision engines, learning models, predictive engines, data mining, Business Activity Modelling (BAM), trending, time series analysis, correlation, regression, tests of significance, pattern detection,probabilities, disparate unstructured source mapping, and the identification homogeneous behaviours is not new. However, now the technology and the data is available these things will be exploited in data analytics for business and research like never before. For a clear example of this consider the idea ‘big data means marketing science’, but the topic of data collection, technological capabilities, privacy, and ethics deserves an article all of its own…

Posted on

The End to End BI Concept

Considering the Full Landscape

Cornerstone Solutions® use an End to End BI approach and here’s why. When thinking about Business Intelligence architecture we need to consider the entirety of the component parts as a whole and not simply individually. In other words consideration is made of the full BI system as a whole and the individual components are not treated in exclusivity of each other. This is because the components will need to act interdependently regardless of whether they have been procured from a single or multiple software vendors.

In this respect the BI system is similar to the human body i.e. everything is so closely related that a felt symptom in one area (a pain in the arm) may be caused by an unseen symptom in another part of the body (a problem with an internal organ). To relieve the pain felt in the arm we treat the unseen causal effect in the internal organ. The same concept of unseen relationship and causal effect applies to the Business Intelligence system. Data flows from end to end through the Business Intelligence system in a similar way to blood circulating in the body and must not be blocked, lost or corrupted at any stage. The BI Architect must ensure this.

End to End BI
The components in the BI system are akin to a linked interdependent chain

Interdependency of the BI system

The early phases of a BI implementation can be usefully considered as akin to those of the Rational Unification Process (RUP) stages of Strategy, Inception, Elaboration, and Construction. A well defined BI strategy is very important. However, perhaps paradoxically the Construction phase is often delivered using an Agile delivery method.

Construction frequently commences with source system analysis, end user requirement gathering and the installation and configuration of the software components. If using SAP Business Information Warehouse, cubes and queries will be developed, or if a relational platform is used a dimensional modelling exercise is undertaken and then the physical tables are developed. The ETL system is designed and developed and reports and dashboards are built.

The relationship between these things is one of interdependency. It’s like a linked interdependent chain. This is why in the implementation methodology of BI System Builders we practice our philosophy of End to End BI. End to End BI takes the view that as each component in the BI system has interaction with and therefore dependency on its related components, BI Breakpoints can occur. BI System Builders take full consideration of the interdependencies in the landscape to ensure prevention rather than cure in their End to End BI project delivery method.