The Dimensions of Data Quality

Data quality is a broad topic, and the data management community has worked hard to dissect and define it so that it’s accessible and usable. In the next few blogs, I’ll provide a deeper look at what is meant by data quality, and examine its component parts.

What is meant by data or information quality? Danette McGilvray, the author of ­Executing Data Quality Projects, defines it as “the degree to which information and data can be a trusted source for any and/or all required uses.” This is a good definition, as it focuses on the outcome of data quality as a practice. It would be counter-intuitive to trust data that was known to be inaccurate, outdated or redundant. Conversely, why would you NOT trust data that has been shown to be reliable and accurate?

Data quality is based on a number of dimensions, which represent different ways to manage and understand the quality of data. These dimensions include:

  • Integrity
  • Accuracy
  • Completeness
  • Duplication
  • Currency
  • Consistency

Data integrity is the most fundamental dimension and the one on which all other dimensions are based. Data integrity is the determinant of whether, on the whole, the data “makes sense” given what is known of the business and its requirements. Data integrity practices include profiling to identify unusual or outlying values, understanding expected distributions, and establishing and enforcing domains of value (i.e., what are the valid vendors of hardware or software).

Data accuracy is a different question than data integrity. A given record may satisfy integrity expectations, and simply still be wrong, e.g. the server is not at location L340; it’s at M345. Data profiling and exception reporting won’t uncover this error; you have to audit the data against reality.

Data completeness is self-explanatory. Is all the expected data there? Are servers or software sneaking into operational status with no repository record? Integrity checks can help, but you may still need some form of audit by testing the data against its reality through a formal inventory process. Reconciliation of multiple data sets becomes an essential tool for supporting this type of initiative.

Data duplication is the flip side of completeness. If a customer has two accounts, then he or she may be able to exceed his or her credit line. If a server appears twice, then the capital budget will be inaccurate. Simple duplication, where two records are identical, can be found relatively easily through basic reporting. The real challenge is what happens when the duplicated records are just different enough to avoid simple reporting? Powerful algorithms such as those used by Blazent can identify potential duplicates, based on fuzzy or incomplete matches.

Data currency is the use of the necessary processes to keep the data current. The data may have once been correct, but not anymore. The server moved, or was decommissioned and sent to salvage. Data profiling over time can identify the “data spoilage” rate, that is, the expected half-life of a data set, and how often the processes will need to run to maintain data currency. Data quality analytics is an important leading indicator for currency issues. The IT Asset system may be behind the times, but the event monitoring system is no longer seeing a feed from Server X – so what happened to it?

Data consistency is the need to replicate data for good, or at least unavoidable, reasons, although it is ideal to have all data in one place. If your IT Asset repository is distinct from your CMDB, then how do you keep them in sync? What about your Fixed Asset and monitoring systems? You might see server data in any of these, with subtle (or dramatic) differences. Reconciling this diversity into a common view is essential, and yet challenging. Detailed business rules may be required to deliver such a consistent, integrated resource.


Now IT is your problem (part 1)

What exactly do people mean when they say “IT”?

  • The computers? The networks?
  • The people who run them?
  • That organization that loves to say “no” and is always slow and expensive?
  • Dilbert’s “Preventer of Information Services”?

IT has been joked about, had its massive failures listed, and had the end of the CIO predicted, and yet they’re having the last laugh.

It’s true that many organizations are decentralizing information technology. What were once called “shadow” systems or “rogue IT” are now the reality, as business units mature through digital transformation. In this evolving context, having a centralized Chief Information Officer in charge of all technology makes about as much sense as having all employees report to the Director of HR.

However, there is a twist. Business executives, many of whom have the perception of IT as “bureaucratic,” may look forward to having their “own” technology and technologists. They may even think they can manage technology better, that it can’t be “that difficult.” Process? Who needs it?

When and if the first few teams are instituted “outside” of IT control, including the ability to acquire computing capacity and build functionality on it, they may move quite quickly, giving their new business sponsors great satisfaction.

But problems soon arise, perhaps as predicted by the CIO when decentralization process started:

  • How are these systems secured?
  • Are they compliant with licensing requirements?
  • Are they backed up?
  • How do you know the data in the system is accurate?

The new business-owned IT finds that some consistent way of building and running new functionality is still required. They may even find that, although computing capacity, developers, and operations staff are directly funded from their budget, enterprise processes and standards represent important guidance that would simply have to be re-invented.

Take for example configuration management. This is the kind of activity business leaders love to hate. At first glance, it might seem far removed from the bottom line. And yet in the digital economy, it’s front and center. Lines of business deploying vast numbers of Internet of Things endpoints have no choice but to inventory and control them; the risk is far too great otherwise. And the infrastructures and services required to do so are not easily or cheaply constructed.

As technology becomes more and more diverse, employing NoSQL approaches that are tolerant of new, diverse data types becomes essential. Understanding security and risk will increasingly require sophisticated analysis across the digital exhaust of these massive, complex infrastructures. At the end of the day, the financial and organizational reporting structures will be less important than the overall digital pipeline. In this new world, data still needs to be governed and managed, auditors will still be evaluating operational practices and risk controls, regulators will still ask for evidence, and will not be interested in whether the organization has centralized IT or distributed digital services.

In spite of this, digital technology will remain with the lines of business, for the same reason that the HR director does not “manage all the people,” nor the CFO “manage all the money.” The Agile movement, as a key enabler of digital transformation, emphasizes that digital products require close collaboration between developers and business experts, along with the fastest possible feedback from the market. The days of the “order taking” CIO organization are over.

What becomes of traditional IT capabilities in this new world? There is a great irony. We have talked for many years about how “IT” must become closer to “The Business.” This is now happening, and now the business is being transformed as much or more than traditional IT.

In this new model, incident management becomes customer service management, configuration management pervades business operations, the Internet of Things becomes the production shop floor, the IT portfolio becomes the digital product portfolio, and IT vendor management simply becomes vendor management, now requiring product specialists in Cloud services. And data management of the digital infrastructure simply becomes data management.

Despite the shift of focus to “The Business”, IT remains the organization that is best equipped to handle the applications once they are deployed. IT sets the bar for security, risk controls and IT Service Management centered around the Configuration Management Database (CMDB).

Digital Enterprise Management – Challenges and Opportunities

We are entering an era where the technology curve we all live on is moving so quickly that the timeframe for development, deployment, and adoption of digital services is now measured in days, rather than weeks or months. The expanding consumerization of IT has vast implications for IT service providers, particularly for those operating at the enterprise level. Over the past thirty years, the technology industry has gone through multiple inflection points: the creation of the Internet, followed by the upgrade to the web; the subsequent expansion of web access to mobile devices; then the rise of social media; and now, the game-changing prevalence of the Internet of Things (IoT). The interval between these inflection points is shrinking, resulting in consistent overlap of transformative technologies. For example, Social and IoT are driving Big Data, which needs cloud-level infrastructure and a whole different level of analytics to be useful. The cool part is that, with the confluence of these capabilities, the entire information technology ecosystem is taking a huge leap forward, driven by the consumerization of digital services.

What does this mean?

First, some context. The term “digital services” seems to be getting tossed around a lot in the technology lexicon, but like the six blind men and the elephant, companies define it to suit their purposes. The definition that we feel best reflects where the market is going—based on extensive conversations with analysts, customers, and partners—is this: Digital services are mobile-first applications that are intuitive and intelligent, bringing contextual information to people based on who they are, where they are, and what they are trying to do. They provide automated self service, crowd-sourced collaboration, and are continuously refined at high speed. The result is a breakthrough in human productivity.

Digital services are automated and controlled by the “customer of the service.” For example, a mobile app (of which there are billions in use) is a service requested and fulfilled directly by a customer with no human intervention, but with real-time performance optimization. This level of service capability—only recently introduced—is already defining an entire generation of end users. The resulting market imperative to manage digital services that align IT capabilities with the people in the business is developing at a blinding pace; our anticipation of this has resulted in the creation and delivery of Digital Service Management solutions, now a core part of BMC’s strategy.

The Big Picture

Digital services are an essential building block for the digital enterprise, and as the management of these services becomes pervasive across a business, we reach the level of the digital enterprise. A digital enterprise is an organization that has a “digital first” mindset for services. Every organization is based on process, and nearly every step of that process is—or can be—automated, which means digital, which means measurable, which means optimizable. Companies now have the capacity to bring the entirety of their capability to bear on every transaction. This is already happening; think about the vast array of processes that are triggered every time you one-click on Amazon. Amazon just announced free, same-day delivery for Prime members, driving an even tighter alignment between process, people, and technology tuned to getting you your shiny object as quickly as possible. The really impressive part is that they do this millions of times per day and, most of the time, it works perfectly.

Amazon is an obvious example, but there are a myriad of digital services that the vast majority of consumers never notice, that completely define their day-to-day lives. IT is charged with the management of these resources and the services associated with them; this includes both existing systems as well as new systems and processes, the integration of siloes of information that are growing exponentially, the maintenance of compliance and governance, and at the same time managing the need to partner with business stakeholders and leaders to drive innovation. For IT, the management of the digital enterprise is the challenge of our times, as well as its biggest opportunity. Digital Enterprise Management is BMC’s vision and architecture for managing and optimizing technology, processes, and policy to ensure continuous innovation and improvement to enable our customer’s digital transformation.

The Internet of Things at Scale

The Internet of Things (IoT) is arguably the single biggest inflection point to hit the Information Technology ecosystem. Ever. We now have the potential to track data on a scale that is hard for humans to imagine. With the release of IPv6 (Internet Protocol version 6) we have a vast increase in available address space; vast enough to assign an IP address to every atom on earth, then do that over one hundred more times. This scale is needed because, up to this point, the majority of online information has been entered by humans; content captured by keystrokes and uploaded photos or videos, with Facebook being the most obvious example. Now we’re entering an era where data is being uploaded by sensors (or smart nodes) automatically, and all of it is designed to increase human efficiency, lower operational costs, and make our lives easier. A few quick examples:

Energy: A San Francisco Bay Area-based energy company is using sensors on oil pipelines to track pressure changes. When a sensor reports a pressure drop, it’s entered as a service request into a Remedy system. Not a typical service desk example, but a great IoT use case.

Automotive: BMW is developing sensor systems that capture driving conditions (e.g. icy roads) and automatically send updates to other BMWs in the area, which then engage the cars ABS system when they approach that patch of road.

Consumer: FitBit trackers are now helping millions essentially gameify their workout routines and compare themselves to their peers. Levi’s recently announced plans to release a line of smart clothing. Nike offers shoes that track your running activity and upload it to your home system for analysis. Amazon is beta testing drones to deliver shiny objects to your doorstep.

Medical: Wi-Fi enabled medical tools and devices are commonplace in most hospitals, and are focusing on both communications between patients and their healthcare providers, as well as real-time monitoring of a patient’s condition. This can also be applied to patients receiving care at home through a wireless connection to their local doctor.

Manufacturing: Any car built in the past few years was assembled by robots, who take parts from a vast ecosystem of supply chain manufacturers and create your sensor-laden automobile. The same thing applies to airplanes, refrigerators, mobile devices, etc. All supply chain components are tracked via IP-enabled devices that report shortages, delays, etc. This was the first industry to really embrace IoT.

IoT is the essence of Digital Service Management, and the implications for IT service providers are game-changing. This is no longer a matter of discovering and mapping thousands of servers or end-user devices on a network. While we’re able to manage data for billions of devices (e.g. mobile phones), we are quickly entering a phase where the number of nodes sending data is rising into the trillions. This is Big Data on a whole new scale, and the ability to track changes that have reverberating effects across a network are going to determine who succeeds and who fails as this technology matures.

In order to succeed in this new era, the Digital Services ecosystem needs to take “industrial grade” services for discovery mapping and change management databases (solutions that tell you what you have and what’s changed) to a whole new level. This includes how the data is tracked, captured, analyzed, and displayed.

Like any other digital service, IoT should be driven by business imperatives. The first step is to determine your organization’s business drivers for requiring data uploads from smart nodes. The second is to make sure you have the right systems in place to support that level of scalability and efficiency, which implies having the right partner to support you.

The Imperative for Digital Services

From the moment we wake up to the moment we go back to sleep, we swim in a sea of digital services. Like most people, my iPhone is the first thing I pick up and the last thing I put down every day. It is with me constantly and I use it nearly continuously—like billions of others. And like billions of others, I take it for granted.

A few weeks ago, I was taking my family to the beach. My son is in the backseat, playing Minecraft with his cousin. There is some heated (nonsensical) debate going on. Nothing unusual in any of this, except that my son is in a car cruising down the California coast and his cousin is sitting on a park bench in London. The level of technology services needed to enable two prepubescent kids to wail on each other with blocky weapons while seated on opposite sides of the world is pretty mind-boggling, and at the same time completely taken for granted. The availability of modern digital services takes the term pervasive to a whole new level, and this presents an excellent opportunity to evolve the existing IT service model to reflect the new digital reality of our lives.

For many years, IT was actually the driving force of technical innovation (even though they often reflexively resist the entry of shiny objects into their domain.) The IT ecosystem has always had to deal with a continuous series of seismic shifts, starting with the rise of the web and quickly followed by mobile, cloud, social, and now the Internet of Things. The protective wall around IT has essentially dissipated under the relentless pressure from the billions driving the consumerization of digital services.

The good news is that this process is forcing a tighter alignment between IT and the lines of business that serve both employees and customers. This is hugely important to IT; expanding from a traditional role of core infrastructure support to becoming an enabler of digital service innovation through the delivery of always-on mobile, cloud, and social-based services, pulls IT into a more strategic framework. Digital services are a competitive differentiator, they have a direct impact on customer satisfaction, and they become the defining context for the overall customer experience across all facets of the organization.

What an end user experiences is the result of the integration of a myriad of systems and processes (e.g. one click on Amazon touches sales, customer support, operations and logistics, shipping, finance, etc.) Every group is updated instantly and all of it coalesces into an experience that the end user barely notices because everything is working the way it’s supposed to. The digital experience that IT delivers touches every facet of a business; all of it is now quantifiable and therefore subject to optimization.

The true core of Digital Service Management is an elegant and compelling experience for the end user, whether it’s a consumer ordering products or an IT service technician answering a help desk query. This is the area where we have been laser-focused; the potential implied in Digital Service Management has fundamentally shifted the tone and direction of our product roadmap and all associated deliverables. It aligns precisely to where our customers are headed, we are actively creating an advocacy framework, and we are delighting our end users. This is the future, not just for BMC, but for the entire IT Service Management ecosystem.

To BYOD or not to BYOD?

While we might claim we want to live our lives like a beer commercial (with a mighty thirst for undying adventure), in the end it’s more puffery than not because we end up migrating toward what is comfortable and familiar. When it comes to our work tools, we don’t want the challenge of the unknown – we want familiarity. That’s why BYOD (Bring Your Own Device) is so popular. We don’t want to jump back and forth between different devices with disparate operating systems, search for buried bookmarks, or explore never-before-seen versions of software. If you’re like me, in a hurry most of the time, anything that slows you down even slightly tends to be pushed aside very quickly.

The Dreaded Learning Curve

When you are handed a completely different and new shiny object at the office, you might feel elated at first (especially if the equipment is better than your own), but you also know you’re going to face a frustrating learning curve that will, at a minimum, slow you down while you figure out the nuances. I heard a lot of friends howling and complaining about being forced from the comfort of Microsoft® Windows 7® into the unfamiliar territory of Windows 8®. Think back to the years before you had a preferred mobile operating system. Before you became an iOS devotee, or perhaps an Android™ zealot. Do you remember how frustrating the learning curve was when you acquired a new phone? It doesn’t matter if it is a massive upgrade – it’s a new massive upgrade, and new nearly always means disruptive.

Pros of BYOD

Is BYOD “good” or “bad”? Like most questions, the answer depends on whom you ask.

1. Productivity

Employers aren’t stupid (most of them, at any rate). They know that when employees are allowed to access work email on a personal smartphone, they will check email 20+ times per day more and end up working as much as two extra hours each day. The Telegraph notes that nearly 9 out of 10 office workers can access work email on their phones, and two-thirds of them check email as soon as they wake up, and right before they go to bed.

2. One Consistent Device

When you use the same devices at home and at work, you don’t need to learn multiple systems or switch back and forth to access bookmarks, search history, apps and software, etc. This is an enormous advantage with BYOD. It is all about productivity, and a consistent experience that keeps me focused and moving fast.

3. Cool factor

Mitch Landry, a BMC principal product manager based in Gig Harbor, Washington said, “Historically, IT shops only would let you use approved devices. It was all about IT. You would go to the IT guys, and everyone was an idiot but them. And if you put something on your desktop that wasn’t approved, they would remove it. That has totally changed. IT is no longer is running the ship, they are a service organization.”

Landry said that a new generation is growing into management positions and demanding to use personal devices (tablets, Macs, smartphones, etc.) once considered unconventional and unsupportable. “They have all these devices and new expectations for service, and the goal for the IT guy today is providing business flexibility and agility. It’s not just about IT anymore, it’s about increasing the productivity of the business.”

Cons of BYOD

1. IT security strains

If you were feeling pretty good about BYOD, not everyone necessarily agrees with you. At the Gartner® Symposium/ITxpo®, Gartner listed BYOD as one of the top 10 strategic technology trends for 2014, and estimated that BYOD will double or even triple the size of the mobile workforce – and place a huge strain on IT and finance organizations.

Gartner describes BYOD as “a disruptive phenomenon where employees bring non-company IT into the organization and demand to be connected to everything – without proper accountability or oversight.” Gartner goes on to warn about BYOD causing violations of all kinds of enterprise rules and regulations, and leading to detrimental impacts on network availability and loss of data. However, with the
 proper governance policies in place, this type of evolution can be handled gracefully. The transformation is inevitable (it’s already well underway), so managing this process is not as dire as the analysts predict. Going mobile with BYOD is not difficult, it’s just complicated, and we do complicated things all day, every day.

2. Indirect costs and threats

According to FireEye, the average enterprise organization is attacked by malware once every three minutes, with each attack costing $3,000 per day or more to recover. Yikes. Opening up the corporate network to rogue BYOD devices increases the likelihood of these costly attacks.

There are security tools and policies that can be enabled on a personal device that allow it to play nicely in a corporate environment and minimize risk, but with BYOD, that risk still will exist. The risk, however, can be controlled, and the associated competitive risk of not mobilizing is far greater.

3. Equipment expenses: Employee costs

Cisco says that 90 percent of employers have some kind of BYOD policy, but the reality is that most of them are not very sophisticated. For example, if your employer doesn’t have some formal process for reimbursement or a way to track the depreciation of personal devices, then the employee bears the brunt of the cost of a BYOD initiative. (Of course, for the employer, this comes out as a Pro.)

Can You COPE With BYOD?

In the wake of the known security perils associated with BYOD, and the obvious desire for employees to personalize their devices, some buzz has arisen around the concept of COPE (corporately-owned, personally-enabled). The idea is to allow the personalization and productivity of BYOD, but with reduced risk.

COPE allows corporate policy makers and IT leaders more control over which devices are supported and what controls are in place on the device, while still accommodating employees who want to personalize their device and content. Just as they could on their own device, employees can send personal emails, access social media, and download photos, but application controls can prevent corporate information from escaping established perimeters. In addition, the IT department controls the device and can remotely wipe the device if the employee loses it or leaves the company.

A Fond Memory of IT vs. Creative

I remember a day years ago when a web designer in our creative group brought his own Mac G4 to the office and managed to elicit a derisive snort from IT. “You don’t honestly think we’re going to support a Mac, do you?” The IT guys didn’t want (or know how) to support a device that wasn’t a PC.

Of course, looking back, I wonder if part of the problem might have been cultural; the IT guys didn’t care for those rogue, black-clad creative folk, slumping in their chairs and listening to bass-thudding techno. They scoffed at the creative types who clicked away their days with design software, sitting at darkened workstations and relishing the perpetual sport of disconnecting the fluorescent ceiling
tubes as soon as the confused maintenance guys popped them back into place. Good times.

These days, even in an enterprise environment, it’s probably more unusual for a creative team not to work on a Mac.

Is BYOD Good News or Bad News?

What do you think? What’s been your experience? And where do you think this is headed? Chime in.

To web or not to web

There has been a running debate for years within the mobile
ecosystem on what approach yields the best result for an end user trying to
access complex information resources. Complex information resources can be
something familiar to most consumers such as on-line shopping, scheduling a
flight, or checking a Facebook status, to something more business-centric such
as adding a new application or workflow to the panoply of capabilities we all
seem to carry around on our mobile devices.

So assuming the objective is to deliver a compelling user experience that encourages
your customers or users to return, then the questions becomes (as noted), to web, or not to web?

To web: we’re basically talking about a web-centric user experience, and by web
I mean a site that is rendered on a mobile device via html 5. There is nothing for
the end user to do but point their browser at a website, and assuming the baseline
software on their device is current, they’re good to go. This is easy, convenient,
and works well for occasional use where the threshold of sophistication requirements
are relatively low. Where it does not work as well is where the process is consistent
that is, routinely transactional), or critical to your day to day (that is, your job depends on
it). Vendors that deliver critical services through an html5 interface are doing their
customers a disservice, their customers are looking for depth, and they’re not providing it.

Not to web: in this instance we’re talking mobile apps. When the requirements get complex, when
you’re dealing with dense data (e.g. tracking sales KPIs as part of a workflow process), or when
you’re dealing within anything in the IT space, such as service management, the app approach
is critical to ensuring the compelling event everyone expects. This is where applications
like BMC’s MyIT come into play; navigating complex information resources should not be complex,
your users have enough to do without forcing them to deal with the limitations inherent in an
html5 interface. Treat your customers with the respect they deserve, and give then a mobile
app that is specific to their requirements.

Defining an enterprise social media strategy

A successful social media strategy begins with just that, a strategy. There is a non-trivial amount of technology at play here, and before wading into the deep end of social media, it is imperative that you have a clear understanding of your objectives. To begin:

1)Set a strategy. What are your long-term and short-term objectives for social media interaction? Are you looking to use a more nuanced social media capability to gain market share, expand against your current user base, or gain a stronger grasp of how to engage customers through a deeper understanding of what they are interested in based on their group dynamic? How are your customers likely to react to more focused attention from you, particularly in a public venue? Does this need socialization prior to implementation? What are your measures for success? How will you know when you’ve accomplished your goals? At what point do you begin adjusting variables, how will you adjust them, and which variables are likely to be most important?

2)Determine the stakeholders. Given the insight social media can deliver to your organization, the number of groups who are going to have an interest is likely to expand from your current operational model. This can include marketing, sales, customer support, merchandising (if applicable), operations, IT, etc. Each will have an interest in gaining a deeper perspective into how to use social media to interact meaningfully with their particular facet of customer engagement, and each is going to have a particular set of data requirements and reporting needs. Get your ducks lined up before your start moving, and once the process starts expect to adjust as you gain clarity into what works and what doesn’t.

3)Study the current level of performance of your existing social media initiatives. How much detail do you have on your existing social media initiatives? Are you able to measure beyond Likes and Retweets? What does the current data tell you? Are you finding a measureable level of success (or perceived success) with your existing initiatives? Helpful hint: before implementing an integrated social media strategy, create a starting frame of reference based on your existing social media initiatives. This will give you something to point back towards (“look how much we improved!”); a fully integrated social media strategy will always have a positive effect on your marketing performance; you want to be seen as the person responsible for making it happen.

4)Examine the current set of variables that can be used to drive segmentation. How many variables are you able to track across different social media groupings of information? The other issue to consider is that there are variables that may be in separate silos that could be incredibly useful for purposes of analysis. Questions you should be asking include:

a)What is the current marketing and/or CRM system of record, what relevant information is in there, and how easy is it to access the data?

b)Where do you track customer support requirements? Customers who are active in social media are already leaving a wealth of details in their wake which can be used to personalize customer support interactions.

c)Are you able to tie social media interaction driven by in-line posts into your customer’s profile? When they receive a communication from you and respond or engage, are you able to capture and track interaction information on the event, and can it be tracked as part of the user profile to driven content optimization?

d)What level of detail can you pick up from mobile social behavior? The beauty of mobile social is knowing when and where to reach out to people in the context of a peer group. A customized and well-timed social communication mapped to the user’s physical location can drive a serendipitous interaction. Don’t just satisfy your customers, delight them.

e)Are you able to assign attribution from social media and map that to your outbound customer engagement strategy?

f)What level of detail are you able to gain from your merchandising system? Do you know who bought what when and is there any corollary data in social media that ties to the purchase event? Look for longer term patterns that allow you to anticipate their next move with high confidence, which can subsequently be driven by social media interactions.

g)How can you integrate this initiative with existing Behavioral Targeting, or Collaborative Filtering or Predictive Modeling, etc. applications?

Social media strategy planning is not difficult, but it is complicated. While the variables that drive social media are constantly shifting, the core elements addressed here are going to be pretty constant, and should provide a consistent framework for execution.

Ad Ecosystem Primer

I’ve been doing some work in the Advertising and Retargeting spaces over the past few weeks. This is a very complex and dynamic domain; lots of moving parts, tons of technology, lots of acquisitions, etc. I thought it would be useful to walk through a definition of who the player are (categorically), since this type of information does not appear to be aggregated anywhere. So here we go:

Ad Exchange Ad exchanges are technology platforms that facilitate the bidded buying and selling of online media advertising inventory from multiple ad networks. The approach is technology-driven as opposed to the historical approach of negotiating price on media inventory. This represents a field beyond ad networks as defined by the IAB (Internet Advertising Bureau).

Ad Networks An online advertising network or ad network is a company that connects advertisers to web sites (also called publishers) that want to host advertisements. The core function of an ad network is aggregation of ad space supply from publishers and matching that space with advertiser demand. The fundamental difference between traditional media ad networks and online ad networks is that online ad networks use Ad Servers to deliver advertisements to consumers, which enables targeting, tracking and reporting of impressions in ways not possible with analog media alternatives.

Ad Operations Also referred to as “online ad operations”, “online advertising operations”, “online ad ops”, “ad ops”, and “ops”, refers to processes and systems that support the sale and delivery of online advertising. These are the workflow processes and software systems that are used to sell, input, serve, target and report on the performance of online ads. Ad operations are typically a department within a digital content publisher, ad network, ad technology provider (such as a rich media vendor or an ad server) or ad agency. They may fall under sales organization, information technology, or may be a separate entity. The primary function of ad operations is fulfilling the order of sale (also called an “ad campaign” or “media ad buy”) purchased directly by or on behalf of an advertiser (also called a “direct marketer” or a “client”). Therefore ad operations is a group that directly responsible for revenue generation.

Ad Server An ad server is a computer server, specifically a web server that stores advertisements used in online marketing and delivers them to website visitors. The content of the webserver is constantly updated so that the website or webpage on which the ads are displayed contains new advertisements—e.g., banners (static images/animations) or text—when the site or page is visited or refreshed by a user. The purpose of ad serving is to deliver targeted ads that match the website visitor’s interest.

Agencies These are similar to traditional advertising agencies (think Mad Men), with the difference being that now they are purely focused on digital media as the delivery method. Most digital agencies can include delivering creative services for banner ads, as well as doing the media buys for larger brands. For example, Samsung introduces a new smartphone and wants to buy advertising across a broad range of publisher sites, targeting different consumer groups with different messages. The process is complex, with lots of rapidly changing variables. It requires specialized competency that a brand like Samsung lacks, but the agency specializes in.

Agency Trading Desk The agency trading desk is essentially a service that helps advertisers and agencies buy online advertising. It is also an advertising technology platform combined with human skills in advertising and technology that provides access to a wildly complex digital advertising marketplace. In terms of customer deliverables, the agency trading desk is a collection of organizational and technology capabilities focused on optimizing digital advertisers’ budgets through real-time bidding and ad exchanges. Within a trading desk, human resources and skills are mainly software engineers, algorithm specialists, analysts and digital media strategists, account manager and buyers. On the technology side, the tools used are essentially DSPs (Demand Side Platforms), APIs, DMPs (Digital Management Platforms) and Ad Servers. Normally the DSP within a trading desk is integrated with Ad Exchanges, SSPs (Supply Side Platforms)and networks. The primary objectives of a trading desk are to optimize buying and campaign deployment according to advertisers’ goals – based on CPM (cost per thousand impressions), CPC (cost per click), CPA (cost per action) and other branding metrics.

Content Delivery Network This (for some reason) is not included in the chart above, but it is an integral part of the advertising ecosystem. A content delivery network or content distribution network (CDN) is a large distributed system of servers deployed in multiple data centers in the Internet. The goal of a CDN is to serve content to end-users with high availability and high performance. CDNs serve a large fraction of the Internet content today, including web objects (text, graphics, URLs and scripts), downloadable objects (media files, software, documents), applications (e-commerce, portals), live streaming media, on-demand streaming media, and social networks. A CDN operator gets paid by content providers such as media companies and e-commerce vendors for delivering their content to their audience of end-users. In turn, a CDN pays ISPs, carriers, and network operators for hosting its servers in their data centers.

Creative Optimization Companies in the creative optimization space focus on businesses that are trying to solve issues related to scalability and measurability of targeted advertising. In most instances, companies are able to identifying their marketing segments, but creating unique ads for each segment would result in an unmanageable ad spend. In addition, these businesses would like to know which elements of their advertising are resonating with which customers, not just base metrics such as CTR. Creative optimization companies provide the tools and services needed to address these challenges, by allowing their customers to bifurcate online ads into their separate elements, and then customize those elements to the individual customer. A travel agency could look at a customer’s geographic location or flight history and suggest trips to locations that will appeal to the user, rather than using generic copy about saving money on flights. A retailer could use the customer’s IP address to identify the closest branch of its store and display the address and phone number in the ad. Creative optimization companies deliver the ability to create copy which changes according to customer data, which means their customers get tailored messages that makes them much more likely to buy.

Data Management Platform A data management platform is the backbone of data-driven marketing, and serves as a unifying platform to collect, organize, and activate first- and third-party audience data from any source, including online, offline, or mobile. A true Data Management Platform should have the ability to collect unstructured audience data from any source, including email, mobile web and app, web analytic tools, CRM, point of sale, social, online video, and other available offline data sources.

Data Suppliers Data suppliers provide consumer-centric purchase and consumption data to help improve and define online advertising targeting by delivering a more detailed and nuanced interpretation of consumer behaviors and habits. Businesses like grocery and clothing stores aggregate shopping behavior and then sell their point-of-sale data to these companies, which interpret and package it prior to supplying it to online retailers and advertisers, which helps them fine tune their product offers and promotions to suit consumer habits and taste. This domain is populated by very large companies, and is one of the core elements of what is commonly referred to as Big Data.

Demand Side Platform A demand-side platform (DSP) is a system that allows buyers of digital advertising to manage multiple ad exchange and data exchange accounts through one interface. Real-time bidding for displaying online ads takes place within the ad exchanges, and by using a DSP, marketers can manage their bids for the display ads and the pricing for the data that they are layering on top of basic consumer profile information to target their audiences. Much like Paid Search, using DSPs allows users to optimize based on set Key Performance Indicators such as effective Cost per Click (eCPC), and effective Cost per Action (eCPA).

Digital or online advertising is a subset of the advertising industry that references electronic communication promotions and marketing. This can include but is not limited to website display advertising (banner ads or rich media advertising), text advertising, search advertising (paid search results), online video advertising, mobile and device advertising (sms, wap display ads, video, application ads), email display ads and text advertising. These advertisements are a forum of revenue generation for content providers.

Measurement and Analytics Refers to companies that track and measure consumer behavior across individual website, networks such as Yahoo and MSN, and includes mobile measurement, social media analytics and a very broad and deep range of online behavior. The companies in this sector are large, very technical, and deeply integrated with their marketing execution cohorts. This sector is probably the closest to the core in terms of how retargeting and ad serving works, since the entire ecosystem depends on analysis of vast amounts of consumer data—this is the measurement and analysis of Big Data.

Media Management Systems (also referred to as Social Media Management) refers to companies that provide customers the ability to coordinate media campaign across multiple channels (such as Facebook, Twitter, LinkedIn, Flickr, YouTube, Google+, etc.) and provides coordination and dashboarding across functions such as publishing automation, ad management, page management, web analytics integration, platform analytics, support for mobile applications, etc. It is, as you can imagine, a very complex domain, and is populated by companies such as Hootsuite, Argyle, Shoutlet, Spredfast, etc.

Media Planning and Attribution Is similar to Media Management Systems, with the addition of a much heavier focus on attribution modeling. The premise they work off is that when brands execute campaigns across multiple channels (including off-line channels) there are influences at play between the channels, and it is important to assign the right attribution to the right advertising element within the sales funnel. As an example, a consumer may see a banner ad on a search return, then subsequently be retargeted as a result of site visit that did not convert, they may see an ad delivered through a set top box, etc. all of which are focused on the same product. Attribution modeling balances out the ad stream in terms of purchase influence. Does credit go to the last thing click prior to purchase, or to the first? If there are multiple stages (and there always are), does credit go equally to all, or are some stages more influential than others? Similar to other technologies in this domain, attribution modeling is complex, algorithm driven, still in an early stage of development, and is tightly coupled to media planning and management.

Retargeting Retargeting is an online advertising technology that serves customized ads to people who have indicated an interest in a brand by visiting a specific website. These users will then see related ads as they navigate to other web sites such as blogs, news sites, or sports pages. Technically speaking, an advertiser places a pixel, or small snippet of code, on their website to begin. This pixel identifies how potential customers interact with their website and allows for segmentation of those customers for later advertising targeting. This is primarily a conversion, rather than an acquisition technology.

Supply Side Platform A Supply-Side Platform or Sell-Side Platform (SSP) is a technology platform with the single mission of enabling publishers to manage their ad impression inventory and maximize revenue from digital media. As such they offer an efficient, automated and secure way to tap into the different sources of advertising income that are available, and provide insight into the various revenue streams and audiences. Many of the larger web publishers of the world use a Supply Side Platform to automate and optimize the selling of their online media space.

Verification and Privacy This covers two areas (hence the name) that are related. Verification focuses primarily on media verification, and includes things such as inappropriate content (don’t server adult themed ads on a page pushing back-to-school sales), as well as management of black lists (sites where ads should never be served), white lists (the opposite of black lists, partner lists, etc. This has also recently expanded to include restrictions on geo-targeting, ad placement above or below the fold, double-serving (same ad twice on a page), fraud detection (including malware, hidden ads, etc.). So verification is that the ad is running as the client intended and nothing that could be subject to misinterpretation is present. The corollary to this is privacy, which includes opt-in/opt-out capabilities, automatic filtering of third party tracking cookies (which is now a browser function), etc. The privacy aspect in particular is getting a lot of attention from Congress, and is starting to have a significant impact on how advertising is delivered.

Big Data Primer

“Everybody complains about the weather, but nobody does anything about it.” Charles Dudley Warner

Similar to Warner’s observation about the weather, enterprises (and the marketers who work in them) are aware of “Big Data” and how important it probably is, but very few enterprises and marketers can articulate a cohesive strategy to leverage big data into their operational initiatives. According to a recent McKinsey report, “Big data: What’s your plan?” the authors note that although “The payoff from joining the big-data and advanced-analytics management revolution is no longer in doubt, most companies don’t have a plan for what to do with that data”.

Given this, it probably makes sense to align on an easy-to-grasp definition of Big Data. Big Data is essentially a reflection of the digital life around us; over a billion users on Facebook pouring their lives on-line, over 5 billion mobile subscribers interacting with both each other and on-line content and services, and the much bigger M2M (machine to machine) data play which is starting to interact with the consumer data play via products such as Google Glass, healthcare informatics, telemetry systems, geo-location targeting, etc. Tracking and operationalizing data on this level requires a different approach to analytics and multichannel marketing.

The upside? Marketers will now have access to unlimited data about their customers.

The downside? Marketers will now have access to unlimited data about their customers.

The challenges associated with Big Data can be categorized along the following sequence:

• Capturing the data
• Storing the data
• Accessing the data in real time for analytic purposes
• Displaying the data in meaningful ways to a non-technical audience
• Operationalizing the data so relevant functional groups can make use of it
• Iterating the use of Big Data in a continuously improving customer engagement cycle

Capturing the data—we live in a multi-channel world. Think about your day-to-day and how you use technology: your mobile phone is probably your alarm clock, can be used to purchase a wide range of products and services, helps you get around (along with the navigation screen in your car), and it can also make phone calls. Your tablet has probably replaced newspapers, magazines, television and possibly books, and we still use laptops when we need to create anything significant. Billions of us move back and forth across these devices without giving it a second thought, and all of these elements are data capture points; anytime a user is interacting with any on-line resource there is data to be captured (and it is). There is also the angle of our moving through a grid network of sensors that track our movements (street level video surveillance being the most obvious example), Wi-Fi hotspots are now expected at nearly any location where people congregate, etc. All this information is streaming in at a stunning rate, all of it comes in via disparate formats, and it all needs to be captured, categorized, and integrated, which goes to the next step.

Storing the data—Facebook is currently uploading 500 terabytes of user data per day . Amazon has commercialized the concept of effectively infinite storage on-demand. Oracle has essentially ceded the mobile market in order to focus on Big Data. Some of the biggest, wealthiest companies in the world are moving aggressively to infrastructure around this trend. How do enterprises open themselves up to this volume of data in a controlled fashion, or more euphemistically, how do you fill a wine glass from a firehose? Technical issues such as data architectures and the budget impact of how to re-align your organization to absorb this much information become critical path decision points.

Real-time access to data-Now that you have zillions of bytes of data at your disposal, what makes you think you can get access to it? Or more importantly, how do you get access to the information you really want, that is, what is the customer doing right now, and how can you leverage this into a sale? How does the system know what data is relevant, and wouldn’t that depend on who is asking the question? Also keep in mind real-time means right this instant, not 10 minutes ago.

Displaying the data—A highly trained analyst can look at the matrix and recognize what’s there, but for the rest of us without advanced technical degrees, how do you visualize data on this level? What does 500 terabytes of data even look like, and how can you extract something meaningful from it? Also keep in mind you will have a wide range of data sets that need to be interconnected if they’re going to be meaningful, and how does that work? What tools do you invest in, and whose needs are going to be dealt with first? Finance, Marketing, and Sales may all be looking at the same data set, but like the six blind men and the elephant, each will interpret and therefor require something completely different.

Operationalizing the data—So you’ve captured it, stored it, accessed it, and looked at it in a way that makes sense. Now what? How can you take this hard-won knowledge and use it to drive marketing initiatives that enable you to provide a truly personalized experience to your customers and prospects? How do you integrate cross-functionally so that customer support and marketing are aligned as a new product rolls out? What tools and processes do you need in order to tie the data together? How do you make this data not only strategic, but more importantly, transactional?

Iterating the data—Then the final step, pulling this data bounty into a closed iterative loop that allows you to fine-tune your execution in real-time in order to optimize your customer’s experience. You, and your customer, and their customers, etc. are in a constant production cycle, whether its cars or cookies or on-line services. Presumably you want your customers coming back for more (as do they for their customers), and also presumably you’re not the only one pursuing your customer (don’t forget the competition, because they won ‘t forget you).