Data Warehousing And Data Pipelines

We Build Data Warehouses

All data analytics concern data and consequently the Data Warehouse in one form or another is always a core component. The Cornerstone Solution® from BI System Builders includes major systems for Data Warehousing. The Data Warehousing systems can be purchased by anyone seeking to build out their own Data Warehouse. The Data Warehousing systems are grouped under three modules, Requirements & Modeling, ETL, and In Production.

Requirements & Modeling

The data model is critical to BI systems because ultimately it will determine if business users can answer their business questions or not.  Cornerstone Solution® systems from BI System Builders always use a ‘pull’ strategy. This means that for the ‘target’ schemas (the tables that the business ask questions from) we always go to the business to understand requirements and model accordingly. Hierarchy MappingTo achieve this BI System Builders subsystems incorporate BEAM (Business Event Analysis and Modeling) techniques developed by Lawrence Corr. These are the most effective techniques we have experienced to gather requirements and design working data models. This does not mean that data modeling becomes easy. Organisations often have unfriendly source system data and complex hierachy structures and  but it does provide a structured and agile way to design. BI System Builders hold requirements workshop with business users and carry out logical modeling in a dimensional way. Business users often enjoy these sessions rather than finding them tedious.

Extract, Transform & Load

Extract, Transform & Load, or ETL for short, is simply the mechanism by which data is ‘extracted’ from source systems (mainframe, spreadsheet etc.), ‘transformed’, that is cleaned, manipulated, summed up etc., and ‘loaded’ into the data models. This may sound simple but the effort and complexity Business Intelligence - Information Consumers involved usually means that the activity will account for 70% of the total Data Warehousing effort. There are often several layers that data must travel through in its journey towards business consumption. The Cornerstone Solution® systems for ETL can be purchased from BI System Builders as standalone modules but we only agree to undertake this after we have validated the existing data models. Weak data models are difficult or or even impossible to load and may result in meaningless or misleading data.

In Production – Live Data Pipelines

Using development environments is one thing and getting working data into production is another. By ‘production data’ is meant trustworthy, working data that is piped and accessible to business users. For this reason the Cornerstone Solution® includes several subsystems enabling ‘code’ in the development environment to make the journey into production. This data is usually loaded into the Data Warehouse every night but can include numerous mechanisms such as real, near real time data and data lakes for mining.  The Cornerstone Solution® In Production Data Pipeline systems include process and governance for the testing and transportation of code and content and the piping of live data feeds into the destination environment for business consumption.

For more information on the The Cornerstone Solution® or any of its subsystems you can reach BI System Builders via the contact page here.

Leave a Reply