Back in 1958, Han Peter Luhn, a researcher at IBM, initiated the concept of Business Intelligence (BI), using the definition from Webster’s Dictionary: to apprehend the interrelationships of presented facts in such a way as to guide action towards a desired goal. Given its definition, Business Intelligence is indeed a vision. It should not be represented by the tools or technologies designed at some given time. In other words, it should be viewed as a company’s strategic vision for transforming data assets into business insights to make data-driven decisions.
Historically, there were two eras that revolutionized and popularized the concept of Business Intelligence. The first was in the 1980s when relational database was invented and became the mainstream tool for data collection and data storage. Database, along with SQL, enabled businesses to access information more quickly to make business decisions based on current facts and historical trends. The 2nd era was in the 1990s when Data Warehouse was born. Data warehouse gathered data from various relational database systems, and transformed and aggregated them further for BI tools to consume, which led to a jump in the accessibility of large amounts of information. As a result, the data warehouse stimulated new technologies that make business users’ lives simpler by allowing them to access more information quicker with better visualizations.
Gaining business insights from data is an iterative process. With a given question in mind, new findings from the data can easily lead to another question. There could be numerous outcomes from a report or dashboard, depending on where you start and what you find. It is typical for a business user to need a suite of reports of various granularities, but still end up requiring data analysts’ help to run custom queries, which could take days or even weeks to get answers. With that being said, an ideal tool should have the following functionalities to satisfy Business Intelligence’s requirements fully:
- Access all data on every level, from the most granular level to the highest aggregated level.
- Access data fast, with response time in seconds
- Do any analysis and reporting at the fingertip without programming
- Share insights with other decision-makers in a common language or visuals, which are both easier to understand and actionable.
There is no single tool that can do well in all the above areas simultaneously. Many companies not only purchase multiple tools to complement each other but also build their reports to fill the gap that those tools cannot do. Also, business and data analysts are very much needed in every company to help the analysis for investigations, troubleshooting, and manually generating the reports or dashboards that a BI tool cannot do.
Recent advances in Big Data, Cloud platforms, Machine Learning, and Data Management have propelled Business Intelligence into the third era with even faster and easier access to information, along with an unprecedented number of choices for visualization and machine intelligence. For example, NoSQL database provides fast access of unstructured data; Visualization tools such as Tableau allow business users to do quick analysis without writing code or queries; Cloud platforms allow streaming technologies and faster data pipelines to publish data from, previously, weekly/monthly to, now, days/hours/minutes. Lastly, Machine Learning helps users from every angle to find the right information, detect outliers, and predict the future.
How can an enterprise fully leverage the new opportunities and succeed in the new era of Business Intelligence? Below lists six key areas that should be carefully thought through with strategic roadmaps to ensure eventual success.
- Know your customers
Identifying your customers and knowing what they want is foremost crucial in making your Business Intelligence journey successful. Generally speaking, there are two types of users:
- Executive and Managers who make data-driven decisions
- Business Analysts who need detailed-level reports to ensure business success
- Marketing and Product Manages who seek new business opportunities
- Data Scientists who use data to run predictive models
- Financial analysts who ensure accurate revenue and profit reporting
- IT Operations who support the production environment successfully
- Data Quality staff who monitor data quality and catch data anomalies
These users’ skills vary drastically, from report analysis, to using Excel, to using a BI tool, and writing SQL queries and programs. IT, therefore, needs to provide different types of data output and tools to tailer different users’ needs. After understanding the customers, a company can then develop the right strategy and tactics to implement business intelligence solutions. An unsuccessful BI initiative often stems from either ignoring the need of a group of users or delivering a tool they do not want, which leads to low customer satisfaction and lower leverage of the data than expected.
2. Standardize Data Pipeline
As stated in my previous article of Big Data Architecture in Data Processing and Data Access, since there could be different BI tools for different groups of users, a standard data pipeline is crucial to ensure a single source of truth. It also increases the efficiency and productivity of an organization by preventing unnecessary data duplications. The last thing any company wants to face is a given measure reported with different numbers from multiple dashboards or reports, and not knowing which one is correct.
Any data, regardless of whether it is collected by an application or from external sources, needs to be cleansed, standardized, and transformed before it is consumed by a Business Intelligence application. Depending on the data sources, there could be multiple standard data pipelines in an organization that may need to be mutually exclusive, with each focusing on its own data sources and responsible data areas.
3. Design the Data Access Layer for Different Use Cases
In the last two decades, the data warehouse has been the favored approach for hosting all useful information in one place for a Business Intelligence tool to access. This approach has several significant limitations:
- A single data warehouse introduces conflicts between processing and data access, which makes it hard to enforce both Service Level Agreement (SLA) of data processing and fast response time of BI reporting at the same time.
- Most data warehouse stores highly structure and transformed data, but not raw or unstructured data.
- Relational database leads to rigidity and high impact when a change needs to be made.
Modern BI tools can now access data from multiple databases or sources, structured or unstructured. Many of them also enable data modeling, data transformation, data profiling, and data sampling for users to determine which data is usable, and to prepare the data within the tool without programming or help from developers. In other words, the data centralization functionality has gradually shifted from the data warehouse to the BI tool, giving users both the capability and flexibility to access various types of data from different systems.
Given the above trend, an organization should carefully choose the right database technology for their BI tools to access, based on the use cases and corresponding data retrieval patterns. The recent advances in NoSQL databases, as well as Big data platforms, offer many different options. Careful testing and Proof-of-Concept is required to make the right technology choice and investment.
4. Empower with Self-service
BI users have always been simultaneously overly reliant on and frustrated by the constant back and forth with the developers who provide the information. Today’s technologies and tools can give users more autonomy to slide and dice the data, look at more granular or unstructured data, investigate data issues, and create reports and stories themselves.
As BI tools consume more and more data, business users will be facing the questions of what data they should choose and where to find the right information. A centralized data management, armed with a robust meta-data management using the data catalog, would solve this issue and become essential for a successful Business Intelligence journey in an enterprise. The benefits are:
- Enabling BI users to find the right data quickly with confidence
- Allowing BI tools to use the same variables, measures and display names in a consistent manner
- Providing a single truth of the semantic layer, therefore, increasing efficiency and productivity across the company
In summary, modern Business Intelligence has given users more power by offering them the ability to find data themselves. This will maximize the chance of users to leverage the data efficiently and productively. It also enables IT to focus its resources on building robust data pipelines and infrastructure and, therefore, delivers scalable solutions to handle more data such as IoT and streaming data.
5. Modernize with Advanced Analytics
BI has been referred to as reporting the past and current vs. Data Science, which is responsible for predicting the future. This is an out-of-date view limited by what previous BI tools could do but is not what Business Intelligence was initially defined. Anything with intelligence should be able to predict the future based on the facts from the past. Therefore, more and more BI tools have the capability to integrate with predictive models developed in other languages such as R or Python, and provide data science features to enable more intelligence out of the box, such as:
- Machine Learning capabilities built into a report or dashboard to automatically detect outliers, trend breaking, and simultaneously send alerts.
- NLP capability to tell the story and conclude automatically from a suite of reports, instead of letting human to look at each report to reach a conclusion.
This is a new but exciting area with high potential for Business Intelligence, which will bring the tools and services to the next level of intelligence for decision making.
6. Monitor Usage for Continuous Improvements
Business Intelligence serves business needs. When the business landscape or the problem to solve changes, a particular report, data board, and data asset can become obsolete. There should be a rigorous process to monitor the usage of the BI tools and their outputs. Any obsolete ones should be retired in an organized fashion. This is important for the overall organic growth within a company to maintain efficiency, cost-effectiveness, and productivity. Any BI tool and products, therefore, should be built with a tracking mechanism from the beginning. For example, it should record users’ access statistics down to the most granular level as which report, which table and which columns were accessed. These stats also give the technical teams ideas of data access patterns so that they can improve the data access performance.
The concept of Business Intelligence came up before relational database was invented. The effectiveness and results delivered by Business Intelligence technologies have been revolutionized twice in the past two decades, and we are now in the 3rd revolution. We sometimes hear the claim Business Intelligence is dying — this is not true. The claim confuses the vision with the tools that had limited Business Intelligence capabilities — any enterprise and company still need Business Intelligence more than ever before. In the wake of this 3rd era, BI is being modernized again with many new capabilities, including more self-service with user-friendly features, data science, and data modeling of predictive ability. Also, more robust data pipelines can be developed to deliver a substantial amount of data that will contribute to new BI tools to drive more intelligence. An enterprise should fully understand its user community, pick the right spectrum of technologies, build a consistent data pipeline, enable data management, and enhance the overall infrastructure, to enjoy a successful business intelligence journey.