Tag: End to End BI

Posted on by — Leave a comment

Avoid BI Breakpoints With Cornerstone Solutions

BI System Builders in their End to End BI philosophy refer to the concept of BI breakpoints. The application of our Cornerstone Solutions® will help you avoid BI breakpoints, but what are they?

BI Breakpoints

Here’s an example of a small breakpoint in a BI system with relational tables that illustrates the escalating effort and associated cost. It starts with a poorly defined report specification. In this example no one identifies the specification as being problematic. So the report is developed according to the specification. However, when the report reaches User Acceptance Testing (UAT) it fails. After consultation with the end user that carried out the UAT it is understood that two important report columns are missing, so the report specification is changed. However, the new data columns are not supported by the dimensional model, so this also must be changed requiring modelling and DBA work. This consequence in turn means that the universe and the ETL packages must be changed and so on and so forth. You can see here the interdependence and knock on effect between the components leading to escalating effort and cost.

However, if you know what to look for you can pre-empt where the BI breakpoints might occur and put quality gates in place. Here are some examples of where BI breakpoints might occur in the dimensional model design of a relational database:

  • The use of a data warehouse generated artificial primary key rather a composite primary key on a fact table
  • Snowflaking in a relational model (not SAP OLAP)
  • A table in which its nature is not made explicit though clear naming convention
  • Table joins leading to chasms or fan traps
  • Fact to fact joins (there are exceptions)

Now a dimensional model design can be tricky to fully evaluate while it remains purely on paper or a computer screen but it is possible at this point to put a quality gate in place to minimise the risk of future BI breakpoints.  Firstly, it can be checked for best practice design principles to avoid the types of things listed in the bullet points above.  Another quality gate can occur after the physical model has been generated. This quality gate is to execute business questions against the tables using Structured Query Language (SQL) and to validate that they can be answered. To choose the business questions to be executed via SQL refer to the report specifications or consult the business users.

It is better to write pure SQL statements using a SQL tool than to use SAP BusinessObjects to generate the business questions in queries. It’s a false economy to wait for Web Intelligence reports to be designed to test the physical model. The reason for this is that the design should be validated as soon as possible, and before the universe is generated. To achieve this some test data must be loaded into the tables but it is far more efficient to identify a design weakness at this stage than later on where changes to the physical model will have a knock on effect on the universe design and any dependent reports.

If it becomes apparent in the quality gate that overly complex SQL has to be generated to answer a business question then the design should be revisited and optimized. The business questions should be focussed around any fact to fact joins, snowflaking or bridging tables as these are potential BI Breakpoints. Their existence can lead to very poor performance, miscalculations due to chasms and fan traps, and the prevention of drill across.

To minimise the risk of BI breakpoints occurring the BI Architect should introduce best practice principles along with quality gates early on. Here’s an example of the best practice principle of designing aggregates. The use of aggregates upfront can address three breakpoint areas before they occur:

 

  • Slow query response times
  • Report rendering times
  • Overly complex reports

It is of course vital to select the correct aggregate type to support the end user requirement. There are several different types of aggregates for example invisible aggregates (roll ups), new facts (ETL pre-calculated fields), pre-joined aggregates and accumulating snapshots etc. Aggregates allow the processing effort to be pushed way from the report and into the database engine and ETL program where it can be more efficiently executed.  However, the use of specific aggregates should not be decided upon until the detailed report specifications are available. Choosing the most appropriate aggregates to support the reports requires skill and experience and is not a menial task.

Aggregate tables can significantly reduce the size of the row set returned to the BI reporting tool. As a rule of thumb it is always better for BI reporting tools to work with smaller row sets. It is always best to force as much processing effort as possible back to the ‘gorilla’ of the BI system – the server itself. It can be even better when processing effort and calculations are forced back into the ETL program. However, care should be taken to educate end users when the effort is pushed back into the ETL program for calculated facts, so called ‘New Facts’, because the facts may be semi-additive or not additive at all.

The downside of using aggregate tables is the increased maintenance e.g. administering ten tables instead of five and the ETL effort required to support them. However, their benefits outweigh their cost.

BI breakpoints can manifest in far more areas than just relational tables, they may be seemingly invisible, and costly when they occur because of the independencies of the BI system. However, applying governance to the BI system around breakpoint areas is not always the most popular notion at implementation. This is because it requires the effort to think clever and to apply hard work, rigour, and discipline upfront to pre-empt problems that are almost invisible. We may be tempted to take short cuts, but, it’s better to apply the extra early effort. Entering into small skirmishes and battles in the early stages to iron out BI breakpoints is much better than allowing them to go unchecked and develop into big fire fights later. Regardless of whether your BI system relies on SAP OLAP cubes and queries or relational tables BI breakpoints are a real threat.

Posted on by

Team

Russell Beech, Founder of BI System Builders & Cornerstone Solution®

I architect data solutions and superintend the solutions that I architect. I build teams that are usually a mix of full-time employees and sub-contractors. I ensure mentoring and knowledge transfer. The emergence of big data has opened the door to a rise in interest in predictive analytics aka machine learning. The technology changes have provided the opportunity to architect data solutions which combine enterprise data warehousing experience with data lake concepts and to apply knowledge of statistics to that data to deliver predictive analytics. To that end I’m putting effort into understanding how the evolving technologies hook in to each other. I have a focused interest in big data technologies especially on the Google Cloud Platform. Technologies such as Hadoop, Spark, Apache Beam, BigQuery and Tensorflow (machine learning) and their integration/virtualization with the enterprise data warehouse.

You can find me on LinkedIn here and subscribe to watch BI System Builders videos on our YouTube channel here.

Posted on by — Leave a comment

The Role of the BI Architect

bi architecture

The Role of the BI Architect

The BI Architect must understand the BI vision and strategy and cause both to come to reality through the end to end BI system. The BI Architect achieves this through fastidious planning and designing pre-implementation, troubleshooting and mentoring during implementation, and governing and evangelising post implementation.

The most effective BI Architects know how to formulate Solution Architecture. They are able to hook together several different architectural disciplines to develop the solution. They will understand Business Architecture: Unless the organisation’s business processes and business requirements are understood how can a BI solution framework be proposed? They must of course be experienced with Data Architecture: Data is at the core of Business Intelligence and the best BI Architects will be experienced data modelers. Hand in hand with experience in data modeling BI Architects must also understand Information Architecture: The sources that the data will come from, it’s journey through the organisation, and how it will be consumed and by who. Experience with Application Architecture is critical for the full understanding of the selection of the BI platform, ETL software, BI tools etc. The most effective BI Architects will know how to size the BI/DW system, understand licensing and pricing and work closely with Infrastructure Architects to understand any required integration and procure the relevant hardware, web application servers, load balancers etc. Finally SLA’s should be considered within the Service Architecture: When must data be available by and how quickly must software execute etc.?

To enable all these things to happen successfully the BI Architect must have a clear understanding of the business process areas as well as a high technical level of expertise in the full BI platform. The BI Architect should have knowledge of the potential BI tools, knowing what they can do, how they integrate with other components and their current development road map. Understanding the development road map enables the BI System to be as future-proof as possible and the BI Architect to advise on the future as well as the now. Poorly informed decisions made today can prove very costly tomorrow. This thorough understanding of the BI tools and components, along with BI processes enables the BI Architect to deliver a BI system that constitutes best practice and exhibits high performance.

When powerful Business Intelligence tools such as SAP BusinessObjects are implemented there must exist the BI function of BI governance. Once the BI System has been developed the best place for governance to be rolled out is through a core BICC and the BI Architect should be central to this activity. For example, consider Self-service BI. When a user interacts with the BI System the user is not simply executing a piece of code within a single application such as Microsoft Excel; several different applications will be interacting together across functional areas. These applications are interdependent and so called ‘stress points’ will occur. In Self-service BI hundreds and even thousands of users can hit the BI System simultaneously.  If governance is not applied to these ‘stress points’ they may become ‘breakpoints’. The BI Architect should be familiar with these ‘stress points’ and in practising End to End BI apply governance to ensure that they do become breakpoints.

Not least required is the ability to evangelise the end user community through clear articulation of the BI processes and demonstrations of the benefits of the selected BI tools. In order to gain end user adoption of the BI System it is paramount to give the very best impression of the BI tool set to the end users. End users need to perceive that they are getting more benefits with the new tools than they previously had. New tools may lead to different ways of working and improved process, this in itself requires some change management. The knowledge and skills of the BI Architect make for an effective evangelist. An internal marketing campaign may be designed to ensure that positive communications about the benefits of the BI system are being transmitted and not lost or watered down in the change adoption process.