Continued growth in AI and automation for data collection

The web intelligence industry has decisively turned to artificial intelligence as the main method to improve its operations.

  • Monday, 20th January 2025 Posted 5 months ago in by Phil Alsop

77% of web intelligence professionals have tried using AI-powered web scraping tools, with 86% stating that it saves time and resources. This is according to new research from leading web intelligence acquisition provider Oxylabs.

In cooperation with Censuswide, Oxylabs surveyed 506 software engineers and web scraping professionals from UK and US companies, gathering insights on the issues faced when collecting web intelligence and the impact it brings to businesses.

Findings from the research confirm that with advances in AI model development, the demand for high-quality public web data is accelerating leading into 2025. 47% of respondents claim to use AI-powered solutions regularly, while 20% have not used them so far but are keen to do so, demonstrating that the web intelligence industry has embraced AI adoption and the benefits it will bring.

Juras Juršėnas, Chief Operations Officer at Oxylabs, said, “Many industries have been hesitant to adopt AI tools, doubting the impact they provide and being unwilling to risk the investment. Yet, AI tools are perfectly suited for larger-scale web data collection. Building and maintaining data pipelines involves lots of moving cogs and an abundance of repetitive tasks that AI can streamline.”

67% of respondents stated that AI will primarily help build data parsers across the web intelligence industry, followed by target unblocking at 59%. Nevertheless, 43% of respondents saw target unblocking as the most suitable for AI applications, as it is a more sophisticated task and a more fitting challenge for the newest technological developments, such as AI tools.

Julius Černiauskas, Chief Executive Officer at Oxylabs, said, “Parsing has remained time-consuming and challenging even when AI was applied, as there have been few exceptional AI-based solutions specialised in data parsing on the market up to now. However, new AI tools such as Oxylabs OxyCopilot are emerging to assist with this. These tools include custom parser builders which enable data parsers to be created in minutes, and request builders to help define scraping requests with only minimal parameters.”

The combination of these two functionalities will help new AI tools to save developers’ time, avoid business costs created by heavily delayed data collection, simplify onboarding for those with less web scraping experience, and reduce spending on internal computing resources.

“The web intelligence industry is a long way from full AI integration, but new innovations are making a major difference in streamlining many tasks. In 2025, web intelligence companies should closely watch how these trends are developing and take steps to leveraging new tools at every opportunity”, concluded Černiauskas.

58% of organizations have observability programs, but 42% still don’t trust their AI models.

Public sector embraces GenAI

Posted 4 days ago by Phil Alsop
GenAI is expected to improve productivity, efficiency, and automation, with long-term ROI for Public Sector organisations.
Survey of 500 tech leaders shows companies are moving fast on AI plans while struggling with scale, integration, and compliance issues.
Insecure cloud configurations create widespread risk, highlighting the urgent need for unified cloud exposure management.

IT teams are losing visibility

Posted 4 days ago by Phil Alsop
New industry report highlights growing SaaS waste, persisting audit costs and evolving priorities as ITAM teams navigate increasing financial...

'Alarming' state of software engineering

Posted 4 days ago by Phil Alsop
Harness, the AI DevOps Platform™ Company, has released a new study revealing a growing disconnect between the promise of engineering excellence and...
New Endava research shows 93% want rapid adoption – even as infrastructure struggles to keep up.
According to a new report from Ericsson, 87 percent of business leaders surveyed in Europe believe 5G connectivity is required to accelerate R&D...