- How are vendors and products selected for inclusion on BI Scorecard?
- Do vendors sponsor this research?
- How are products evaluated?
- How long have you been evaluating BI products?
- Who thought of the scorecard approach?
- Do you have any biases?
- What other conflicts of interest might affect objectivity?
- How frequently are evaluations updated?
- How is your research different from other analyst firms?
- Do you take briefings from smaller vendors?
1. How are vendors and products selected for inclusion on BI Scorecard?
Vendors are selected based on a combination of market share and mind share. We cover leading vendors that customers routinely inquire about. When we cover a niche vendor, it’s because we think their product is compelling or complementary to a BI deployment.
2. Do vendors sponsor this research?
No, although they may subscribe to the web site.
3. How are products evaluated?
Vendors routinely brief us on their products. In addition, we discuss product capabilities with customers, partners, and consultants. Finally, we test the products hands-on based on an evaluation framework driven by customer requirements. We also rely on a network of experts to validate findings.
4. How long have you been evaluating BI products?
Cindi Howson has been evaluating BI products for more than 20 years and has implemented several of the products reviewed (and some products not yet reviewed here), both as a customer and as a consultant. The content on this web site represents intense research since March 2003.
5. Who thought of the scorecard approach?
Wayne Eckerson, former director of research for TDWI, recommended we use the scorecard approach when we first began co-teaching Evaluating BI Toolsets at TDWI. I resisted, thinking scoring products would introduce too much subjectivity and vendor contention. The hands-on approach has helped to minimize subjectivity and ensure the products are evaluated consistently. As for the vendor contention, well, clearly the yellow and red scores can be contentious! However, if I can’t figure out how to do something or the feature is not well documented, then I’m skeptical that customers will be able to take advantage of the functionality either. Bottom line – end customers have been enthusiastic about the scorecard approach! The side-by-side comparison is now available at both summary and detail levels.
Over the years, the scorecard approach has been refined to apply weighted averages to the summary scorecards.
6. Do you have any biases?
We prefer a buy versus build approach as industry research suggests this is more cost-effective, successful, and sustainable. In this regard, we look for out-of-the-box capabilities. Products that are poorly documented and vendors that do not respond to inquiries in a timely way, make the evaluation process more challenging.
7. What other conflicts of interest might affect objectivity?
Much has been written about conflicts of interest between analysts and vendors (see here and here). For this reason, we actively ensure the majority of our business comes from end customers, either through subscriptions or consulting engagements. Our work for vendors is generally limited to educational events that allow us to interact with end customers such as webinars and presentations at user conferences. Vendors may engage with us directly to present at a webinar, but more often, this is facilitated by TDWI, The Business Intelligence Network, or Intelligent Enterprise.
Vendors have sometimes inquired about the book BusinessObjects XI: The Complete Reference. Interestingly, customers have never expressed a concern about this. Did the vendor pay me to write the book and does it make me biased in favor of BusinessObjects? The publisher, McGraw Hill, hired me to write the book, and they maintain editorial and production control. We are intimately familiar with all the BI tools, but it’s up to the publisher in terms of what books they see a sizable market, without competitive titles. Also, it’s important to note that the Complete Reference books are about how to use a product, and not whether it is the best tool for a customer. Given that authors get between $1 and $3 for each book sold, writing books about any BI product is more of a service to the BI community than a financial motivation.
In order to minimize other potential conflicts of interest, BI Scorecard employees do not own stock in any BI vendor nor do we resell software.
8. How frequently are evaluations updated?
Our research is continuous. We generally publish one new report a month, update multiple data points in the detailed scorecard monthly, blog bi-weekly, and provide webinars periodically. The detailed scorecard is updated based on vendor briefings and demos which is sometimes based on beta product, Overview reports and detailed evaluations are finalized following hands-on testing. Timing for these evaluations is dependent upon the vendor supplying us with software.
9. How is your research different from other analyst firms?
Our evaluations are based on hands-on testing so are more objective, consistent, and detailed. As our experience is originally as customers, we also view things with a pragmatic perspective and real-world requirements in mind.
10. Do you do briefings with smaller vendors?
Yes! However, as a small company, we have limited bandwidth to cover new vendors and are more likely to do briefings with vendors that our customers have asked us about or that have a clear differentiator in the marketplace. When we do briefings, vendors should be advised that we may take screenshots for note-taking purposes, blogs, our course materials. Any items that are confidential should be clearly labeled as such.