Data Source Consolidation
Four Core Data Source Consolidation Considerations
Every organization depends on a range of data sources to support different operational initiatives, and as we all know, these data sources are either additive, or continuously expanding, changing, merging, etc. The dynamics of managing and integrating data sources are complex and highly variable, and often result in having new sources which are similar to or duplicate parts of existing sources.
The challenge is that these disparate data sources must work together and share information. In most instances, the sheer number of sources and their data can’t keep this from happening without adding significant complexity and effort. The continuous expansion of data sources is a key driver for consolidation initiatives, and sources that can be configured to work together can improve organizational efficiency and reduce capital expenditures.
Document all Data Sources in Scope
The first step to execute any consolidation initiative is to document and assess all the data sources within the scope of the initiative. It is not only important to understand what each source contains, but also how various teams use it. For example, some teams might consume and use the data source as-is, while others may aggregate and normalize it with other data prior to use. Documentation of the data sources must also include any integrations, which might need to be eliminated or re-established with a different source once consolidation is complete.
Evaluate Their Use
When all the data sources are identified and documented, it is important to evaluate each for its intended use. This means assessing the data sources for expiring licenses, retiring technologies or strategic plans to move from those tools and their respective sources of data. This type of evaluation minimizes disruption to users from repeated changes to the sources they leverage on a daily basis. It also offers potential opportunities to work jointly with already budgeted initiatives to help with the consolidation.
Planning and Scheduling the Consolidation
The next stage is planning and scheduling, which will help determine the sequence in which the data sources are consolidated. Integrations and their interdependencies can cause the sequence of consolidations to become very complicated, which implies it may be necessary to re-sequence the order temporarily to keep operations flowing smoothly. If this is done, it is important not to invest too much in temporary measures since they must be moved again to the permanent source.
Analysis and Comparison of Data Models
The last stage of preparation before a data consolidation project begins is the analysis and comparison of data models and structures. The data sources will not have identical models, and defining the mappings between them will require a considerable amount of effort. Before beginning, costs, additional time, and increased complexity of data manipulation due to structural differences must be considered. It is also important to factor in the long-term implications of maintaining these manipulations if there are no plans to use technology to automate them. The cost of automation technologies may seem high until you compare them to the cost of doing the same thing manually.
Stagger the Data Source Consolidation
Once the analysis and planning are complete, the actual consolidation of the data sources can begin. It’s important to be prepared for the consolidation to occur in a staggered manner, since there are always delays due to some data sources not being ready for moves or changes. There will also be some data sources whose consolidation plans must be expedited because of external dependencies. The consolidation plan must be flexible, but disciplined, since it is the nature of this type of effort to be complex, interdependent and subject to unexpected changes.
In Summary
Consolidation efforts are a continuous requirement due to the massive organizational and operational burdens associated with the constant expansion of information sources and the data within them. The costs of not maintaining this pattern of growth are unsustainable; efforts to consolidate data sources and simplify operations are not easy, but can reap significant savings and increase operational efficiencies. In the short-term, it may appear too complex to execute this kind of initiative; however, with the proper assessment, planning and use of available technologies and services, the return far outweighs the challenges of maintaining the status quo.
Blazent’s Data Quality solutions provide organizations with the flexibility to retain all existing data sources and still benefit from consolidation by intelligently merging multiple sources. The process Blazent creates delivers consolidated records that are complete, accurate and current.