Managing data pipelines continues to present significant challenges for organizations, with an overwhelming 70% of respondents rating pipeline management as 'somewhat' or 'extremely' complex according to a recent data integration and AI report.
According to the Data Integration and AI-Readiness Survey commissioned by intelligent data integration platform Matillion, more than two thirds of respondents are struggling with pipeline management.
The survey revealed an alarming picture of data integration, an industry facing ever-increasing demand thanks to the growing business need for artificial intelligence (AI) and generative intelligence (GenAI).
Matillion CEO Matthew Scullion said: “Data engineering is boring, gritty and repetitive. Data teams are wasting valuable hours on low-value build, maintenance and management. Instead this time could and should be spent building valuable data products that can be used for new business impacting initiatives.“
The survey highlighted a substantial productivity drain with 64% of organizations reporting that their data teams spent more than 50% of their time working on repetitive or manual tasks.
Rather than increasing staff (and increasing the overheads associated with that), organizations need to identify data integration solutions that empower their current teams to work more efficiently.
Scalability emerges as another critical concern, with 89% of organizations noting issues with their current data engineering platform’s ability to scale pipelines to meet data processing needs.
Scullion added: “This survey highlights the need for a unified solution that is accessible to a wider scope of the organization - easing the load on data engineers and data teams, while empowering business leaders to make data-driven decisions. Bringing AI into this mix creates an incredibly compelling vision of data engineering of the future, where data engineers are able to focus their time on innovation and doing what they do best - solving business problems, rather than manual, laborious pipeline management.”
This Data Integration and AI-Readiness Survey was conducted in partnership with Perspectus Global in January 2025. Respondents included 307 data decision-makers and data user titles, based across the UK and the US.
