Collaboration around data is holding back 95 percent of enterprises deploying AI

Nearly 90 percent of organizations investing in AI, very few succeeding.

Tuesday, 31st July 2018 Posted 6 years ago in by Phil Alsop

A survey commissioned by the leader in unified analytics and founded by the original creators of Apache Spark™, Databricks, has revealed that only one in three AI projects are succeeding, and, perhaps more importantly, it is taking businesses more than six months to go from concept to production. Organizations are hindered at multiple stages of the process when bringing AI projects to production. According to 95 percent of European respondents, collaboration between data engineering and data science teams is a challenging issue. Nearly all respondents cite data-related challenges when moving AI projects to production. IT executives point to unified analytics as a solution for these challenges with 90 percent of respondents saying the approach of unifying data science and data engineering across the machine learning lifecycle will conquer the AI dilemma.

The research, commissioned by Databricks through CIO/IDG Research Services, shows that more 93 percent of organizations surveyed are investing in technology to help with data prep and data exploration/modeling, including data processing, data streaming, machine learning and/or deep learning tools. As a result, organizations are using an average of seven different tools for data prep and modeling. Based on these results, it is not a surprise that European organizations cited technology as their most common obstacle when moving AI projects to production.

Additional results of the survey speak to the complexity and organizational confusion being creating as companies pursue AI initiatives:

The average number of AI projects considered completely successful within enterprises is around 39 percent according to those surveyed in Europe, compared to an average of 35 percent of projects in the US.
98 percent of the surveyed believe preparation and aggregation of large datasets in a timely fashion is a major challenge;
96 percent of respondents found data exploration and iterative model training challenging;
90 percent cited the deployment of models to production quickly and reliably as a significant challenge

David Wyatt, Vice President and General Manager EMEA, Databricks, commented on the survey’s results: “Getting AI right is challenging, and one of the biggest hurdles to success is how teams collaborate around data. The research shows how difficult and time consuming it can be to turn raw data into valuable insights for the business. With unified analytics, enterprises can bring together their people, processes and technology to deliver results faster – not only does this make projects more efficient, it increases the chances that these projects can succeed in meeting their objectives over time.”

So, what will help these organizations conquer the AI dilemma? The surveyed executives said they need end-to-end solutions that combine data processing with machine learning capabilities. These streamlined solutions would simplify workflows, improve efficiency and ultimately accelerate business value.

In fact, nearly 80 percent of executives surveyed said they highly valued the notion of a unified analytics platform. Unified analytics makes AI more achievable for enterprise organizations by unifying data processing and AI technologies. Unified analytics solutions provide collaboration capabilities for data scientists and data engineers to work effectively across the entire AI development-to-production lifecycle. With more than 90 percent of large companies facing data-related challenges and increasing complexity driven by an explosion of machine learning tools, the need for platforms and processes that can remove technology and organizational silos is more pronounced than ever. Unified analytics provides an ideal approach for companies facing modern AI implementation barriers.

Databricks accelerates innovation by unifying data science, engineering, and business. Through a fully managed, cloud-based service built by the original creators of Apache Spark, the Databricks Unified Analytics Platform lowers the barrier for enterprises to innovate with AI and accelerates their innovation.

Databricks commissioned IDG Research to conduct analysis of the AI, machine learning and deep learning landscape in large enterprises. The survey looked to sample the experiences of those working in specific senior data engineering and data science roles at companies with more than 1,000 employees. The audience of 200 people was split equally between the United States and Western Europe.