Envision a city that has built a new light rail system as an additional, and much needed, form of public transportation. It’s been running for a while now, and seems to be popular, but of course the transportation initiative doesn’t stop with people boarding a train. Ongoing, strategic analysis is essential to measure the performance, usage patterns, and the impact the new system is having on key goals that city and transportation officials have established.
Similarly one must extend a business intelligence (BI) solution, once it has achieved a certain degree of maturity. Suppose your BI environment has been up and running for a while now, but you don’t really know how it’s performing. In many ways, this lack of knowledge is tantamount to undertaking a journey with your eyes closed.
Obstacles ahead? You can help prevent them, if you keep an eye out and adjust course. Not many organizations do this, though, as the following graphs from a Forrester online survey reveal. (These graphs appear in the Forrester BI report “BI on BI: How to Manage the Performance of BI Initiatives.”)
Percentage of organizations that are measuring their BI solution
Source: Forrester’s Q1 2014 Global BI Maturity Online Survey (total of sample size = 37)*
Based on these results from Forrester BI research, there’s certainly room for improvement. Here are some examples of why, how, and what to measure when evaluating the “intelligence” of your organization’s BI solution and assessing the tangible business value of BI.
Reasons why the performance of your BI program should be measured on a regular basis include:
- Justifying costs and investments that have been made, or that are being proposed.
- Figuring out what is and isn’t being used, so that you can course-correct.
- Fine-tuning workflow/processes to support goals of the organization.
The Forrester BI report “BI on BI: How to Manage the Performance of BI Initiatives” recommends*:
- Figure out what you need to measure. Fewer is good, so focus on metrics that are most crucial for your needs. Forrester analysts recommend tracking no more than 10 to 15 metrics.*
- Consider tools or methods that can also provide quantitative information; you can get qualitative information from methods like surveys or observation.
- Investigate the capabilities of your own platform to see what kinds of metrics it can provide.
- If your BI platform doesn’t have adequate capabilities for measuring its own BI, look into what others (such as third parties or system integrators) offer in this area.
The following are some examples of questions and areas to consider—both quantitative and qualitative (with potential overlap between the two)—as you measure your established BI solution:
- What’s the adoption rate/number of users of the BI platform or tools in the organization?
- What are the usage patterns of specific tools, capabilities, data warehouse, in-memory, CPU, and memory? How many are using a given tool or capability? How are these being used? How many queries are there?
- What’s the ability to achieve business results and enable timely decisions? Can non-technical users quickly produce reports and dashboards, and other content? What is the average time?
- What’s the average response time of the database and server for a defined task?
- How much can you use prior technology investments, such as in something like SQL Server Analysis Services? To what degree are people taking advantage of these prior investments?
- What sorts, and what volume, of dashboards, reports, and other content are being created?
- How many people are using mobile BI?
- Is the solution agile, not overly complicated, and flexible, yet governed?
- How scalable is it?
- How secure is it?
- How relevant is the content that’s available?
- What’s the level of data integrity and accuracy?
- To what degree does the solution lend itself to collaborative BI?
Seeing is doing
Part of intelligence involves adapting to the environment you’re in, and adjusting what you can control, to achieve maximum benefit. So take off the blinders, and look closely at the environment around you. What you see might surprise you. But whatever discoveries you make, you have a much better chance to course-correct and make your BI solution smarter by assessing your BI maturity than if you’re on autopilot. Ignorance is not bliss!
*BI on BI: How to Manage the Performance of BI Initiatives, Martha Bennett and Boris Evelson, December 22, 2014.