DataOps: bridging the divide

The digital economy has been defined by the proliferation of computing through mobile, web, and IoT, all enabled by tech such as DevOps and cloud. But if the past is about speed, the future is about data. With the rise of connected consumer technology and advances in machine learning and artificial intelligence, data, and the insights we can derive from it, are the foundation of tomorrow’s economy.

Despite the obvious value, companies struggle to realise the opportunity lying dormant in their data. When developers, testers and data scientists have access to the data they need, innovation happens faster. But data is more complex, more unwieldy, and harder to secure. Legacy tools, organisations, and processes can’t resolve this tension - we need a new approach.

DataOps is an emerging cultural movement that aligns people, process, and technology to support high velocity collaboration and innovation through data. By focusing on bringing people together to secure, manage, and deliver data, DataOps practices can enable data to flow wherever it’s needed by the business.

But what actually is DataOps?

Like DevOps, DataOps is broad in its approach across people and technology. DataOps is about aligning tools, processes, and culture to facilitate secure access of data, regardless of how data may be used across existing teams or organisations. Its goal is to improve outcomes by facilitating access to data and collaboration between the teams that use data.

DataOps brings together everyone that is involved in the creation, management, and use of data. This includes those that operate data infrastructure such as IT, DBA, and InfoSec teams. But it also includes those that use data to drive new projects and innovation, such as developers, testers, analytics, and data scientists.

If these teams exist today, why do we need a new approach to bring them together?

The problem

The rising demand for data creates natural tension with the forces trying to manage cost, complexity, and security. This tension creates a divide between the teams and tools trying to provide people access to the data they need. Higher quality data drives better outcomes, and this divide can force compromises that put the business at risk. For example, if a developer needs access to the latest production data to test a new change, their demand creates friction on the systems and processes required to fulfil that request. If that request takes days or weeks, the team is forced to decide between moving slower, or risking poor application quality due to inadequate data.

This problem is only getting harder. The rise of artificial intelligence and machine learning increases demand for large, representative datasets. Cloud and DevOps brings elastic application environments to the masses but struggles to fill them with data, and the proliferation of voice, image, and video data require new tools and techniques.

The solution

DataOps is as much about transformation of people as it is about transformation of technology. To be successful, companies need to tear down the walls that have been created over time and instead take a data-centric view of their enterprise, one that brings together everyone that manages, or needs access to, data.

Start by mapping out the data you have (or want to have), and where it’s needed. Think through not just application development, but analytics and data science, archive and compliance, and cloud migration. From those data needs, articulate the policies that need to be in place to ensure that only the right people get access to the right data in the right environment. For example, what data is subject to HIPAA or GDPR? What data can the public cloud support versus a private data centre? 

By bringing everyone together into the conversation and focusing on the business policies and needs of data users, companies can map out how data should flow across their organisations and infrastructure. This will expose constraints and points of friction that can be mitigated or worked around through new technology or processes. DataOps solutions should see to put self-service access to data in the hands of users, while empowering IT teams with confidence that data is flowing according to the policies they’ve established.

A happy ending with DataOps

Global revenues from artificial intelligence (AI) for enterprise applications is projected to grow from ?1.14 billion in 2018 to ?22.3 billion in 2025, with 49% of companies in 2017 using some form of marketing automation. However, your AI is only as good as the data you feed it, and much of the most importance data remains trapped behind traditional enterprise systems and processes.

With DataOps, operations can focus on establishing and securing all the data sources needed by the project teams, without being bombarded with a plethora of requests for access from all over their organisations. Meanwhile, developers are equipped with fresh and secure production data which can be easily refreshed, reset and shared. The result is faster projects, of higher quality, at a lower cost.

It’s time to stop thinking about digital transformation and start embracing data transformation through DataOps.