Enabling Big Insight with Big Data (Trends and Insights)

Article
07/18/2014

Big Data is at the forefront of the minds of many business leaders, in different domains and industries. In addition, with the widespread hype about Big Data in the media, many businesses have begun looking to spearhead technology initiatives with Big Data features.

When considering such initiatives, it’s important to keep in mind that Big Data is a tool, not a task. When planning, an enterprise must define and focus on business objectives and outcomes that will determine the success of the technology initiatives.

This post, and several future ones, are based on an upcoming white paper called “It’s not about Big Data, it’s about Big Insight” by Achim Granzen, Architect in the Data Insights Center of Excellence.

The following contributors and reviewers provided input to the white paper from which these posts were drawn: Ken Collins, Fidan Boylu, Philip Reilly, Benjamin Wright-Jones, and Delbert Murphy.

The Data Evolution

In less than a decade, data and information have moved from the back-room of enterprises to become critical business assets. Data that was previously locked away in structured applications is now made available everywhere, in every form and format.

Across every industry and every organization, data and information have become “first class” citizens, producing new knowledge and insight. New technologies and techniques add significant capabilities to mature Business Intelligence (BI) and Business Analytics solutions for enabling enterprises to obtain business insights.

While most of the current definitions of Big Data are descriptive, a more encompassing definition includes the potential value proposition as well, as in the following description:

“Big Data represents the trends, technologies, and potential for organizations to obtain valuable insight from large collections of structured, unstructured, and fast-moving data.”

The New World of Data

Traditional BI uses data from internal business transactions and operations. The infrastructure for supporting such data includes Enterprise Resource Planning (ERP) systems, data warehousing with extract-transform-load and query & reporting capabilities, Online Analytical Processing (OLAP) analysis, and data mining.

In such an environment, data may have attributes such as:

Scarce
Expensive
Easy to manage
Structured
Manageable format and structure
Primarily internal

These attributes typically describe a system of records (for example, customer records).

The traditional BI model does not go away—it is enhanced and expanded as technology progresses, for example adding self-service BI capabilities for business users. Traditional BI will continue to be the approach of choice in certain business domains, such as managing corporate financial performance.

However, the traditional BI model and technologies are not well-suited to use semi-structured or unstructured data, especially in case of the data originating outside the boundaries of an organization.

For example, strategic business insights can be derived from audio/video/images, text/xml documents, web logs and click streams, social media postings, data from sensors/devices, and event and streaming data.

In this new world, data has attributes such as:

Plentiful
Cheap
Difficult to manage (with traditional techniques)
Detailed but of questionable (unknown) quality
Nearly infinite variety in format and structure
Often external

These attributes typically describe a system of interactions or engagements (for example, interactions with customers).

Traditional relational databases are not well suited to handle such data. With the arrival of Big Data technologies and techniques, it is now possible to bring these data sources into the decision-making process in a financially feasible and timely (often near real-time) manner. Some examples are:

Analyzing physician notes to help discover new cures for diseases
Analyzing social media postings from millions of users to determine active trends
Analyzing streaming data from thousands of sensor to diagnose the operational status of manufacturing facilities in near real time
Analyzing hundreds of thousands of images to find anomalies that could be threats to public safety
Analyzing life sciences data (genomics/proteomics) to discover patterns in cell behaviors

The following illustration shows how the traditional BI model is expanded to include the new data types, new sources, and new technology capabilities commonly grouped under the term Big Data. Traditional BI is represented in green; Big Data elements are represented in blue.

Figure 1. Expanding the traditional BI model in the new world of data

Data Leadership

This new world of data has implications throughout the entire organization. Besides the technology impact outlined in the previous section, it impacts an organization’s culture, processes and people. All of these dimensions have to change and adapt, to enable an organization to succeed.

Processes

Business Analytics is an agent for change; ideally it results in a constant loop of asking questions, gaining insights, taking action, measuring results, then asking new questions to get new insights.

Processes in an organization have to adapt to this cycle of constant change. This also affects the way business departments and IT collaborate (or more often than not, fail to collaborate).

The constant insight loop requires a constant and close collaboration to ensure that all undertakings are driven by business questions and result in achieving a set of business objectives. Technology is the means to achieving these objectives.

People

With the now all-popular and ever present Data Scientist taking center stage in any discussion of new roles and skills required to tap into the value of data, the need for close alignment with the business might well justify a more to-the-point description as a “Business Scientist”: a Data Scientist with sufficient business domain expertise to provide the “missing link” (for example, a Marketing Scientist).

Such a role is not an IT or technology role anymore, but solidly anchored in the business, and the key skills are not just about academic qualification, but particularly on the application in a business scenario.

Culture

All of the above can only really happen in a culture that fosters sharing and insight, essentially a data driven culture. Organizations are adding C-level data roles, in the form of a Chief Data Officer (CDA), or Chief Analytics Officer (CAO). These positions recognize the value of data as a strategic asset of the organization – a major cultural change for most organizations.

Defining Business Objectives for Big Data

IT departments these days are often tasked to “start with Big Data;” however, without a clear business objective, such endeavors rarely lead to success. Rather, initiatives should focus on a concrete business objective such as Customer Insights (or Citizen Insights), Predictive Maintenance and Management, and so on.

Two important scenarios (to be discussed more in the next posts), are based on Big Data are Customer Insight and Operational Insight.

Customer Insight

Customer Insight is one of the prime areas when looking at business scenarios impacted by the evolution of data. It is also a prime example of how an entire business has to change in order to reap the benefits of that evolution, as customer behavior, communication channels and buying mechanisms have irrevocably changed over the past 5 to 7 years. As a consequence, the way organizations market and sell has changed already, but the new complexity also comes with the potential for new and deep insight previously neither possible nor feasible.

Operational Insight

With the rise of the Internet of Things – connected sensors and devices for just about any type of equipment, from phones to household items to heavy machinery – it becomes possible to not only monitor equipment in real time, but to use predictive analytics to identify potential disruptions and malfunctions before they occur, thus saving huge amounts of money on defects and service disruptions.

Summary

Starting with an understanding of the desired business outcomes is imperative for any undertaking which aims at obtaining insights in the new world of data. New technologies and techniques, along with disruptive market trends, provide the playing field for exciting new ways of running, managing and changing an organization. Analytics is the agent for change, enabling new insights, leading to new ideas, leading to new questions – a perpetual cycle of insight and action.

Figure 2. Perpetual Analytics Cycle

Integrating a perpetual analytics cycle is definitely not something that can be solved by just new technology – it is an enabler, but not sufficient. As outlined earlier, the new world of data requires people with new skills, new ways of looking at business process, and a new culture that values data and insight based decisions making. In addition, the benefits of analytics can only be reaped in a truly collaborative environment of business functions and IT functions.

Figure 3. Four Critical Dimension for succeeding in Analytics

Analytics has the power to change an organization (uncovering new business areas, new ways of doing business, new products and services), therefore managing that change across the four critical dimensions People – Processes – Technology - Culture is the key for succeeding in Analytics. In upcoming posts, we will discuss Customer Insight and Operational Insight scenarios as examples.