Tag: BI Architecture

Posted on by — Leave a comment

Cornerstone Solution Q & A

BI System Builders

We are asked many questions relating to the BI System Builders Cornerstone Solutions® and here are the answers to some of them.

 

What is a Cornerstone Solution®?

A Cornerstone Solution® is an End to End best practice framework for Business Intelligence and Data Warehousing. It has been developed by BI System Builders Ltd. It has two key components. The first is comprehensive Solution Architecture and the second is a delivery model that incorporates detailed project planning. It is designed to facilitate a set of Business Intelligence services to be brought together in a BI methodology that results in measurable deliverables, mitigating risk, and maximising project success. It can be thought of as a strategic framework for BI Architecture and Delivery.

 

Is it just for SAP BW systems or can it be used on other platforms?

A Cornerstone Solution® is vendor agnostic. It is not reliant upon any specific technology and is fully configurable.  BI System Builders have expertise with SAP BusinessObjects and its connectivity into SAP BW and performance and historically, have partnered with the vendor SAP. However, SAP BusinessObjects is a technology, a BI Platform, BI tools and Data Integration capabilities. A Cornerstone Solution® is not tied to a specific platform such as Netweaver, a data integration tool, or any software vendor for that matter.

 

Is your method only relevant for SAP BusinessObjects technology, we want to use a different BI software vendor?

A Cornerstone Solution® is not software vendor specific. A Cornerstone Solution® is a generic end to end Business Intelligence framework rather than a SAP BusinessObjects method. It exists to deliver world class Business Intelligence and Data Warehousing solutions. We have referred to SAP BusinessObjects frequently because it’s a powerful BI platform and it’s also historically where a lot of our core technical expertise lies. However, if SAP BusinessObjects was not the technology of choice it would not matter, a Cornerstone Solution®could and has been applied to other technologies\vendors.

 

Is a Cornerstone Solution® applicable to Big Data?

Yes, a Cornerstone Solution® includes thinking about data governance, data quality and data integration as well as architecture, process and analytics. It would be no problem to incorporate capabilities such as learning models into the Cornerstone Solution® framework and massive amounts of data require more governance not less.

 

When Russell Beech was employed by BusinessObjects he co-authored a white paper entitled ‘The Rules of Engagement’, is a Cornerstone Solution® not just a re-hash of that?

No, a Cornerstone Solution® is much more than anything that the ‘Rules of Engagement’ could attain to. The ‘Rules of Engagement’ was written specifically to help Professional Services (BusinessObjects field consultants) implement the BusinessObjects Analytical Applications successfully on large scale deployments. It was based on experience of troubleshooting complex implementations in large organisations but a Cornerstone Solution® is much more. A Cornerstone Solution® is a complete End to End BI strategic framework that includes architecture, process and governance. The philosophy is essentially to architect and to plan so as to avoid problems rather than to let them grow to the point where troubleshooting is necessary.

 

Does a Cornerstone Solution® include ETL?

Yes it’s designed to deliver an End to End BI solution and data integration is a critical component. The Cornerstone Solution® considers everything from business analysis and requirements to source system data. Intrinsic to the solution is the ETL system itself.

 

Why do you say that Business Analytics are the end game of a Cornerstone Solution®?

A Cornerstone Solution® will help your BI project run smoothly. However, it can do far more than that. The full purpose of the Cornerstone Solution® is to use BI to help an organisation grow, increase profitability, gain competitive advantage, and increase efficiency and effectiveness. Business Analytics provide key information to business users helping them to achieve their purpose. A Cornerstone Solution® actually starts with corporate strategy and objectives. It is the cognizance of these that drives the BI Architectural decisions. Raw data has relatively low value unless it can be turned into usable information. The BIDW architecture must fully support this and the information is usually consumed through some form of analytic.

 

Where did the term ‘BI Breakpoints’ come from?

A career as a BusinessObjects consultant often leads to facing challenges. As consultants become more senior and experienced they are invited to troubleshoot the harder problems. We have seen many of these harder problems. The term was coined by us to describe the problems or failures in the BI system. These failures can include everything from physical components such as faulty software installs to activities such as ineffective requirements gathering leading to data models that are not fit for purpose. Understanding BI Breakpoints and pre-empting where they are likely to occur is core to implementing a Cornerstone Solution®.

 

If it’s a delivery methodology why do you say that BI strategy is also a component?

A Cornerstone Solution® has a defined framework. The application of some of the Business Intelligence services within the framework leads directly to tangible deliverables e.g. developing ETL code. The development of code is a tactical manoeuvre within the big picture.  However, the framework itself is part of a still bigger picture, that of BI strategy. Formulating the BI strategy is critical to getting the tactics right for successful delivery. So a Cornerstone Solution® is a strategic BI framework rather than just a delivery method alone.

 

How is the Cornerstone Solution® different to what other BI companies do?

Perhaps other companies do exactly what we do? It’s possible. But it seems evident that many BI projects get into trouble. If a Cornerstone Solution® framework was being applied we believe that there would be less project difficulties.

 

What do you mean by ‘Cornerstone’ and why do you say that it’s a solution?

A cornerstone is the stone that forms the base of a corner of a building that joins two walls. It is the first stone to be laid down and on which everything else is built. The cornerstone in a successful Business Intelligence system/solution is the Business Intelligence framework. In a Cornerstone Solution® everything is built out from this framework.  Solutions are ‘acts’ or ‘ways’ to answer problems. Cornerstone Solutions® are a framework facilitating a set of Business Intelligence services to be applied to help the organisation solve its business problems through best use of its Business Intelligence system.

 

How is a Cornerstone Solution® different to just program or project management?

A Cornerstone Solution® includes comprehensive project management and a good project manager is critical. However a Cornerstone Solution® includes BI strategy, a defined framework, and predefined matrixes, templates, and artefacts. The customer may often provide the program management with our support.

 

If you write about Cornerstone Solutions® other consultancies will just steal your ideas?

It’s often said that “imitation is the sincerest form of flattery”. We put many helpful things on our website at bisystembuilders.com and are fully aware of what we are doing. We like to give things away and will publish more and more. But reading about something and talking about something does not necessarily imply the knowhow of how to actually do it. The name Cornerstone Solution® is a registered trademark.

 

Can the Cornerstone Solution® work for a small company?

Yes, absolutely yes, and cost effectively too. A Cornerstone Solution® is scalable. It includes everything required for a big undertaking but some components are superfluous to small projects. The Cornerstone Solution® is designed in a modular fashion enabling us to strip out unnecessary components but leave the End to End BI view intact. Smaller companies only pay for what they need; they do not pay for unnecessary ‘nice to haves’.

 

Can BI System Builders help us with small parts of our BI project or do you only do full BI implementations using a Cornerstone Solution®?

BI System Builders can help you with small parts of your project.  A Cornerstone Solution® is modular and we select the best practice part that is applicable to what you want us to do for you.

 

I’m new to Business Intelligence and some of the things that you talk about with Cornerstone Solutions® seem complex. Where can I read more about BI

You can read more at bisystembuilders.com and we plan to add lots of other content too. For an informed read on Business Intelligence we also thoroughly recommend the book e-business Intelligence authored by Bernard Liautaud the founder of BusinessObjects. It was published back in 2001 but is still very useful.

 

This is not really about Cornerstone Solutions® but what does the BI System Builders logo represent?

Actually it does relate to Cornerstone Solutions®. The logo is in the shape of the end of a large key. A key in itself is a solution to unlock something. You’ll notice that there is also a flow through the key. This is symbolic of the flow of data into information through your business… through your BI system. BI System Builders have the key that unlocks the flow. The key is a Cornerstone Solution®.

 

Why do you say that a BI project doesn’t need to be like a walk through a jungle when you talk about a Cornerstone Solution®?

Walk through the jungle with no map and no guide – not a good idea, it’s a big place and easy to get lost and end up in trouble. You know you could get caught out by things that you failed to see or expect, you could become disorientated, lose energy, and lose spirit. So you could consider one of the following:

  1. a) Employ a guide but take no map – if you have a good guide then you don’t need a map. However, you are trusting the guide to be who he claimed to be, but what if he’s not authentic? On the journey how will you know if you’re getting lost? When you’re really lost and come to the conclusion that you’re following a charlatan!
  2. b) Take a map but no guide – it’s possible, but anyone embarking on that journey should also take a compass and be sure that they can use them both effectively together, as well as knowing the jungle dangers and having wilderness experience.
  3. c) Take a map and a recommended guide – you get the best of both worlds. The guide can keep you safe and at the same time show you exactly where you are on the map so that you can check progress.

If BI System Builders is like a recommended guide, then a Cornerstone Solution® is like a map. You can know where you have come from, where you are, where you are going and where you will end up, safely!

Posted on by — Leave a comment

Business Intelligence Solution Architecture

If your business intelligence initiative or program is struggling it may be because you do not have effective BI solution architecture in place. BI solution architecture involves far more than just understanding a technology product stack. As its name suggests it is by nature a solution and it encompasses business, information, application, infrastructure, data, integration, and service architectures.

In this new video I show you how to design business intelligence solution architecture that will make your business intelligence initiatives successful. I use SAP BusinessObjects as my example but the concepts are actually vendor agnostic so you can use the ideas with numerous BI and data technologies.

Watch the video

Posted on by — Leave a comment

End To End BI 2

THE ENVIRONMENTS – BI SIZING & HARDWARE (Staging) –
In the previous article we gave an introduction to the End to End BI Philosophy and introduced the various layers that comprise the Business Intelligence Data Warehouse (BIDW). In this article we will focus on getting your End to End BI environment ready by making consideration of sizing and hardware with some reference to software. This article assumes that you are a Business Intelligence Consultant or Architect with design authority or a BI Programme Manager or Business Owner wanting to quickly gain insight into Business Intelligence. If you are technically orientated, your decision making will benefit significantly if you collaborate with an Infrastructure Architect – so go make contact with one in your organisation.

You should plan to size and have hardware procured for five exclusive environments. These are a sandbox, a development environment, a system test environment, and pre-production and production environments. Let’s take a brief look at the nature of these environments.

Business Intelligence - Information Consumers

The sandbox is your playground; it’s a place where you can experiment and learn with zero risk of compromising a deployment on a box used to maintain real content. You can install and configure software, write code and make and break the environment. During my time working for BusinessObjects many software engineers preferred to use VMware for their sandbox. They would install and configure the BI environment and then clone it. Any system corruption – no problem, just blow out the corrupt environment, clone your clean environment again and your back in business. Ensure that the sandbox has sufficient specification to support the minimum requirement of the platforms you intend to install.

The other four environments are where you will manage the lifecycle of your Business Intelligence content. Your development and system test environments are a cut down version of your pre-production environment. Development and system test should run the same software versions and patches as the pre-production and production environments. However the CPU and RAM is usually considerably lower.

The development environment is the place for authorised developers to develop and unit test their code and content. It is not a place for so called ‘ghost’ developers to experiment. Ghost developers are persons that gain access to a development environment with a view to ‘experimenting’ and learning how things work. They will do this anonymously using a generic logon, backdoor or someone else’s password! If you’re a ghost developer, and you know if you are, go play on the sandbox and don’t risk compromising another developer’s hard work on the development environment. At this point you will benefit if you have version control software in place.

After unit testing content on the development environment it is promoted to the system test environment. This is the first time that the testing team will have the opportunity to test the code. To test effectively the system test environment must be stable. Any content identified with defects is passed back to development to be fixed. When content passes testing it is promoted to the pre-production environment. In pre-production we expect high volumes of industrial strength data to be available for the first time. This data may need to be encrypted/ anonymised depending on your industry. It is the opportunity for the content, solution and system to be pressure, stressed, volume tested (PSV) before being unleashed on the production environment. It is a useful place for DBAs to monitor query execution plans and performance tune

Your production environment should mirror your pre-production environment and be secure. By passing content on pre-production testing you are saying that the code is fit to go through the Change Advisory Board (CAB) and move to production. Testing on pre-production is only truly legitimate if the pre-production environment is exactly the same as your production environment.

For brevity the remainder of this article will now focus on the pre-production environment but the concepts can be applied successfully across all four environments. Taking the pre-production environment as our worked example we will need to consider the staging layer, the data mart layer, the ETL system, the Business Intelligence Platform, the RDBMS, hardware and software. This means that there are implications for budget, procurement timelines, licensing, disk space and performance. You may also need to consider contractural terms with third party suppliers of source data and any outsourcing organisations that manage your source systems. If your organisation utilises an ODS or BCNF entity tables you will also need to make other additional considerations. You must now make sizing estimates that will allow you to make informed decisions.

Our first consideration is the BIDW layer known as the staging area as this is where we will (usually) initially land the data. Let’s first of all consider the staging tables. This is the place that you will bring your data in to from outside the BIDW, it’s a kind of landing area. Remember that your sources for Business Intelligence may be wide ranging and include applications such as mainframe legacy systems and an ODS. Your historic load will probably comprise very large data volumes. You will need additional storage space if you need to store the data in flat files etc. before loading to staging. This can happen because of licensing, operating and contractual limitations.

There are three things to consider regarding the architecture and sizing of your staging area. These are the data volumes and nature of your transient staging area, your persistent staging area, and any data archiving area. Your transient data is that data made available to the ETL system for transform and load. You will not want to lose your transient data because you may need to reference it at some point in the future. However, for ETL performance reasons you will not want to hold unnecessarily large volumes of transient data so you will move the data off to your persistent area once the transforms are successful. You must make two decisions. The first is for how long you need to hold your transient data and the second is for how long to hold your persistent data. Let’s say that for legal and audit reasons you decide to persist your staging data for five years. At the end of the period you will need to delete or archive the expired data. To make these decisions you need guidance and remit from senior stakeholders.

It’s time to start making some calculations. Here’s an example thought process. One column field of type integer will equal four bytes. Now let’s say that my staging table has sixty columns that are all of type integer, that’s sixty multiplied by four equals 240 bytes of data per single row. Now let’s say that the source system serves up 60,000 rows of data per night. That’s 60,000 * 240 =14400000 bytes. Ok and let’s say now that I’ll hold my transient data for seven days, that’s 14400000 * 7 = 100800000 bytes of data. Let’s convert that to GBs. 100800000/1024/1024/1024= 0.093877 GB. Let’s say that I have 50 staging tables of similar size loaded every night. That’s equal to 0.093877 * 50 = 4.69GB. Not a lot of space by today’s standards. Now let’s calculate for persisting. That’s 4.69 * 52 (weeks) * 5 (years) = 1219.4 GB or a 1.2 Terabytes of data. We have just calculated your incremental load data for the next five years. You will probably also want to load the historic data held in your source systems. So you must repeat the calculations for the number of years back data (historic) that you want to make available to business users.

So now I have an idea of what disk space I need to procure for staging. Here your Infrastructure Architect can advise on the standard for storing data on a SAN etc. Where possible you should be working within the mandate and standards laid down by the Enterprise Architect.

There are always more details to consider around these areas but the things we have discussed form the backbone of what you will need to think about. There isn’t ‘one size fits all’ but there are frameworks and patterns which repeat themselves again and again. Your considerations now are the procurement, licensing, and installation of any additional/ new RDMS platform and the procurement of any additional hardware to make the staging area performant. In the ideal world these would already be specified in the Solution Architecture, but the reality is that BI Solution Architecture is often overlooked.

The next article in this series will consider your pre-production data marts.