Cloud integration: shining a light on dark data

For most, cloud environments such as data lakes and data warehouses are not new concepts—on average, 60% of data lives in a data warehouse with the other 40% living as dark data. The catalyst? Users without access to live data take it offline and run analysis locally, removing control and visibility from the rest of the business (e.g., spreadsheets or forecasting documents), creating an additional knock-on effect of data governance.

The key to overcoming these barriers is to move and shape the vast volumes of data into one centralised location, creating one source of truth. This allows businesses to extract insight, helping inform business decisions at speed, as recently shown by ESPN.

The nature of sports broadcasting is competitive in itself. User experience is paramount to avoid subscriber churn. ESPN needed a way to quickly aggregate customer comments to spot emerging issues with ESPN access and identify which services were affected. This was initially a slow process as there was a lack of visibility into existing activity, which was holding back any decisions that would improve the experience. For example, ESPN would need to refer to reports made after a sporting event, usually provided via PowerPoints and Excel documents, sometimes taking weeks or months to create. It was only from there that decisions could be made on how to improve based on customer feedback.

ESPN took it upon itself to streamline this process, knowing that online viewing of sporting events is only going one way. It knew it was crucial to provide customer service in real-time, utilising call center activity and social media platforms to monitor fan interaction. To do this, Domo and RXA developed and deployed a solution capable of extracting keywords and information, analyzing sentiment, and summarizing everything that’s been pulled from various channels for sharing with ESPN’s customer care teams. To further aid viewers, ESPN now pushes out notifications through AI chatbots, FAQ alerts, and live-site announcements. As a result of these efforts, over the 12 months ending March 7, 2020, customer satisfaction was up by 9%, and customer self-service by 200%.

So how can your business get there?

The first part of this process is to determine where the ‘dark data’ resides. Dark data takes shape in many forms, from offline excel spreadsheets to Salesforce pipelines to monthly profit and loss statements. Gaining insights from this segregated or siloed data can take hours, if not weeks, of data analysis and reporting. Senior leads will often call upon departments to produce these reports individually before dissecting it and finding common themes to inform their next steps. This process is slow and naturally affects consistency across the business, as many departments find themselves singing from different hymn sheets.

The solution is to integrate and automate how people see data across the business, providing one consistent line of truth for all departments. This integration happens by plugging all ‘dark’ siloed data platforms into one data visualization platform. At ClearScore, this process helped the data analyst teaming more efficiency field questions like “how much

money we made yesterday” or “how many users we signed up last week.” This shift in data management allowed its team to move away from time-intensive reporting toward larger data science initiatives that have an impact on the bottom line, such as building credit risk or eligibility models in developing markets.

Democratising dark data and hosting it under one roof allows all departments to foster a culture where decision-making happens without going through IT support. Business leaders should focus on driving this behavior change, helping department leads innovate in how their teams are deployed and operate, using live data as a tool for creative and impactful decision making.

While democratising data is crucial, certain datasets may need to be governed or only seen by certain team members. However, data integration in the cloud doesn’t mean losing control, as each user of the central platform can be assigned a role or attribute—meaning they have certain access levels from the moment they log in.

An example of this is Domo’s Dynamic Personalized Data Policies (PDP), which makes it possible to create policies based on user attributes instead of individuals. This tool allows a business to scale policies with growth without editing entitlement policies. Business can take scaling a step further through augmenting built-in roles to maintain governance policies, which still champions the use of one unified platform and ensures only certain members can export and share data.

Whether it be shielding the IT team from endless data requests or activating chatbots via AI to speak with your customers, the possibilities of uncovering and using dark data are endless. Once a data-driven culture is created across your business, you are only bound by the creative use of its application.