Data Integrity: the key to operational insights or the elephant in the room?

Throughout history, business has always struggled with the challenge of data accuracy and integrity. While there is clear operational and strategic value in accurate and dependable data for decision-making, the operational cost of achieving and maintaining data integrity can be a substantial barrier to success for many organizations. As IT and OT (Operational Technology) systems evolve, legacy data is continuously migrated to new systems. Examples includes off-the-shelf software and SaaS solutions which come with their own taxonomies, or data models which are merged with home-grown systems. As he needs of the business change, the maintenance of any level of data integrity can easily become cumbersome and costly.

Executives constantly ask their IT leaders how they can improve the quality and integrity of data in order to obtain the insights needed to guide their company effectively. While it sounds reasonable, it may well be the wrong question. Rather than focusing on the quality of raw data, a better approach is to focus on the quality of insights available and the speed/cost to obtain them by asking, “How can we better leverage the data we already have to cost effectively obtain the insights we need?”

Advances in machine learning, data science and correlation analysis during the past decade have enabled a broader range of capabilities to analyze data from disparate operational processes and information systems. This has been accomplished without developing some of the structured relationships and incurring data-model-integration costs associated with traditional data warehousing and reporting approaches. Keep in mind that modern analysis methods are most appropriately suited to gaining operational insights and do not (presently) replace the structured data required for financial reporting and regulatory compliance.

Through assessment of the trends and relationships between different data elements, modern data analysis systems are able to “discover” a variety of insights that may not have been available during the past. Examples include undocumented dependencies within operational processes, sources of data inaccuracy and the evolution of operational processes during time. Instead of focusing on what is “known” about operational data, modern methods focus on understanding what is “unknown” about operational data. It is the variances, exceptions and changes to data that yield true operational insights. This type of analysis not only yields higher value decisions, but also can be accomplished at a lower cost to the IT function, because it does not require the “scrubbing” and “integrating” of operational data that was once required. In many cases, poor data quality and integrity can yield higher quality insights in areas where traditional approaches masked operational challenges and opportunities.

The key to operational insights is actionability, where you can answer questions like:

• “Where are inefficiencies and time delays occurring in my operational process?”
• “Is there variation in performance across operational functions/locations?”
• “Where are inconsistencies being introduced into my operations?”

None of these questions focuses on “Where are my operations performing as designed?” but rather asks, “Where are my operations doing something unexpected?” By focusing on the unknowns and unexpected behavior, decision makers can identify actionable areas either to mitigate cost/risk or to exploit opportunities for value creation.

Is data integrity the key to operational insights or is it the elephant in the room? That depends on how organizations want to view the situation. Modern data analysis methods and improvements in technology enable what once was viewed as a challenge (or liability) now to be used as an asset to drive opportunity. Data Integrity at both the informational and operational level is a core requirement of any modern business, and has been an area of focus for Blazent since the early days of Big Data.