It will probably stay that way unless people start to approach enterprise BI search differently than they do ordinary Web search. While end users still need the same simple user experience, enterprise BI search introduces a new level of complexity because of the heterogeneous architecture of applications and data stores.

Organizations need to plan for more than an ordinary Web search, which primarily addresses the reading and parsing of presentation-oriented file formats and URL-based site crawling. Too few search vendors stress the importance of accessing and indexing the 300-plus enterprise applications and databases that can make or break a BI search solution.

Defining the Scope of Enterprise BI Search
Misunderstanding or intentionally limiting the breadth of an enterprise BI search solution can lead to incomplete solutions, inappropriate vendor selection, and - ultimately - compromised user experience.

Search solutions must answer all relevant questions, whether they are about detailed data or summaries. Hence, the scope of BI search extends along a continuum from unstructured documents and aggregate reports to individual records and transactions stored in applications and databases (see Figure 1). While a solution can be implemented in stages, selected technologies should enable indexing and reporting along the entire continuum. This simple approach helps map vendor capabilities and determine which offers the best fit.

Unstructured BI-Relevant Content
Web search engines like Google, MSN, and Yahoo! crawl file directories to find and index 300 or so file formats, including presentation-oriented and unstructured files such as HTML, Word, Excel, PDF, images, and multimedia files. These unstructured documents affect BI search implementations because they provide context and substantive detail to reports; for example, court documents often supplement arrest records. Thus, generally speaking, more file format support results in a more complete index with less document preparation. While Web search engines must crawl directories blindly, without prior knowledge of the stored file formats, this feature is less important for enterprises because most standardize on content-creation tools and document formats.

Three hundred file formats will usually suffice, but consider whether you will need an engine that can integrate proprietary parsers for unsupported file types. Most BI content - especially the structured content - can be transformed into an appropriate indexing format. Since few search engines offer robust transformation tools, BI vendors can fill this gap.

BI-Specific Content: Reports, Records, and Transactions
Reports and transactions are BI-specific content types. Their original formats don't really matter, because they can be transformed into, say, HTML or XML for indexing by Google. More important is the need to access data sources and applications in order to extract and enrich data, making the information meaningful for a natural language search. Specialized search engines have started to develop access and integration capabilities, but only BI vendors currently provide enterprise-level capabilities.

Reports - static aggregations of individual transactions - are stored in report libraries or file systems. Search engines can index reports independently or with BI vendors in the same way they index any other unstructured document. The lack of context makes it difficult to distinguish, for example, one profit report from another among the hits on the search results page.

BI companies provide value by supplying metadata in the search results that the end user can use to identify the most relevant report. An integrated BI and search solution lets users retrieve reports, refresh the data, and modify the report content - important capabilities when up-to-date reports are required. Only BI vendors can generate entirely new reports from the hits, such as what users would need while searching for inventories that might be out of stock.

Most BI vendors only index reports. While it's tempting to think that users don't need anything more, most questions are about the details of individual records and transactions, especially so in operational BI. Experts estimate that 80 percent of enterprise data is structured and that, from a decision-making point of view, the value of structured transactional data far exceeds that of unstructured data. That implies that enterprises should focus on indexing structured data first; unstructured content is misconceived as low-hanging fruit because it was the core competency of search engines.

Search engines significantly expand BI query capabilities in this area. BI companies use structured queries to find or filter data in known data sources using known parameters. Search allows users to find data not only in structured (dimensional) fields but also in unstructured (CLOB or text) fields without prior knowledge of the data sources or the parameter values. Thus, customer records can be retrieved by names in structured fields or by customer clues recorded in the free-form text fields. Some BI companies provide the missing link through transactional indexing, which includes data access and metadata enrichment.

Transactional Indexing
Search engines can rarely index transactional data without pre-processing and enrichment - what search companies call "content aggregation" - because the raw data isn't suitable for natural language query. For example, users search for products by names and descriptions rather than inventory numbers, so they need more than the data from a star schema's fact table. At a minimum, indexing this content requires supplemental look-up values (natural-language descriptions) for all keys and codes.

Transactions can be enhanced by appending data from other tables, databases, and applications, or by pre-aggregating records. Help desk applications, for example, create a new entry for each communication with a customer, and relate it to a customer case using a reference key. Indexing each communication record separately will create fragmented search results; not indexing all customer communications will create an incomplete record for searching.

The solution requires enriching the incoming record with the available customer information, re-aggregating all communications into a single indexed message, and passing it to the search engine to replace the previously indexed record.