Survey reveals “massive productivity drain” in data engineering

Complexity, scalability and compatibility remain challenging - 70% data workers struggle with pipeline management.

  • Tuesday, 22nd April 2025 Posted 1 month ago in by Phil Alsop

Managing data pipelines continues to present significant challenges for organizations, with an overwhelming 70% of respondents rating pipeline management as 'somewhat' or 'extremely' complex according to a recent data integration and AI report.

According to the Data Integration and AI-Readiness Survey commissioned by intelligent data integration platform Matillion, more than two thirds of respondents are struggling with pipeline management.

The survey revealed an alarming picture of data integration, an industry facing ever-increasing demand thanks to the growing business need for artificial intelligence (AI) and generative intelligence (GenAI).

Matillion CEO Matthew Scullion said: “Data engineering is boring, gritty and repetitive. Data teams are wasting valuable hours on low-value build, maintenance and management. Instead this time could and should be spent building valuable data products that can be used for new business impacting initiatives.“

The survey highlighted a substantial productivity drain with 64% of organizations reporting that their data teams spent more than 50% of their time working on repetitive or manual tasks.

Rather than increasing staff (and increasing the overheads associated with that), organizations need to identify data integration solutions that empower their current teams to work more efficiently.

Scalability emerges as another critical concern, with 89% of organizations noting issues with their current data engineering platform’s ability to scale pipelines to meet data processing needs.

Scullion added: “This survey highlights the need for a unified solution that is accessible to a wider scope of the organization - easing the load on data engineers and data teams, while empowering business leaders to make data-driven decisions. Bringing AI into this mix creates an incredibly compelling vision of data engineering of the future, where data engineers are able to focus their time on innovation and doing what they do best - solving business problems, rather than manual, laborious pipeline management.”

This Data Integration and AI-Readiness Survey was conducted in partnership with Perspectus Global in January 2025. Respondents included 307 data decision-makers and data user titles, based across the UK and the US.

AI is now the leading security concern

Posted 2 days ago by Phil Alsop
AI surpasses ransomware as the top concern, as organizations navigate the double-edged sword of innovation and risk.

Workforce crisis sparks debate over HR & IT merger

Posted 2 days ago by Phil Alsop
New study of global tech leaders finds IT leaders believe combining functions could boost productivity and engagement.
Cyware survey identifies significant gaps in internal collaboration, tool integration, and automation — with only 13% confident their systems...
Seventy-seven per cent of engineering leaders identify building AI capabilities into applications to improve features and functionality as a...

Data streaming enables AI product innovation

Posted 6 days ago by Phil Alsop
In the largest global report on data streaming, 89% say DSPs make AI adoption easier.
73% are investing in AI-specific security tools with either new or existing budgets.
Public sector organizations recognize the potential of AI for enhancing decision making, improving service delivery and driving operational...

AI calls for cyber resilience rethink

Posted 1 week ago by Phil Alsop
Unveiled at the RSAC™ Conference, the 2025 LevelBlue Futures Report finds only 29% of executives are prepared for AI-powered threats, despite...