Making non-linear linear

There has been a fair amount of recent coverage on the shortcomings of business intelligence as the concept starts to move out of the purview of Fortune 1000 companies and into more mainstream usage. One area referenced consistently as an area needing work is the BI community’s steady focus on structured data at the expense of unstructured data. Business intelligence as an application suite is still relatively nascent in its deployment; while it is widely used by large companies (although not consistently or comprehensively), the vast majority of businesses are not Fortune 1000, and wouldn’t recognize business intelligence if it hit them in the head. The opportunity here is that with no prior frame of reference, there is a great opening for BI vendors to step in with a solution that is ideally geared towards the requirements of the SMB market. Two core drivers here are 1) use of unstructured data as a BI feed, and 2) dumbing down the application as much as possible so mere mortals can feel comfortable using the product on a transactional level.

Most data (over 80%) in most companies is unstructured. E-mails, narrative reports, legal documents, any product centric information (data sheets, functional specifications, etc.) is unstructured, and it’s where the majority of mission critical information exists. There is a huge inventory of information just sitting there, beyond the reach of BI or analytics engines because most BI apps are designed to think in a linear fashion, and unstructured data is by definition non-linear. You can add metatags or some form of XML structure to your documentation (which is finally starting to happen), but this also pre-supposes some sort of referential taxonomy to organize the information once it’s been made ready to be pulled into a BI application. The people who are most likely to be transactional users of this type of technology are not trained to think in terms of a taxonomy, this is generally a luxury that only large companies can afford. So that is one area that would need to be addressed before there is broader market acceptance of sophisticated business intelligence applications.

This leads to the second requirement; make this thing easy to use. If you’re like most of us, your day-to-day work keeps you running at full tilt. Stopping what you’re doing to run up a long steep learning curve is probably the last thing you want to do, yet that is what most BI vendors expect of their end-users. The more you can shield your end-users from the innards of the technology and provide them with a simple, graphical, drag and drop interface, the more likely they are to adopt a system that minimizes a trip outside their comfort zone. This is another, potentially fatal sin of BI developers: “we’ve developed a highly sophisticated analysis product, let us show you”, when what they should be saying is “what type of information do you need to do your job better, and how can we make it as simple as possible?”