Continued growth in AI and automation for data collection

The web intelligence industry has decisively turned to artificial intelligence as the main method to improve its operations.

  • Monday, 20th January 2025 Posted 3 months ago in by Phil Alsop

77% of web intelligence professionals have tried using AI-powered web scraping tools, with 86% stating that it saves time and resources. This is according to new research from leading web intelligence acquisition provider Oxylabs.

In cooperation with Censuswide, Oxylabs surveyed 506 software engineers and web scraping professionals from UK and US companies, gathering insights on the issues faced when collecting web intelligence and the impact it brings to businesses.

Findings from the research confirm that with advances in AI model development, the demand for high-quality public web data is accelerating leading into 2025. 47% of respondents claim to use AI-powered solutions regularly, while 20% have not used them so far but are keen to do so, demonstrating that the web intelligence industry has embraced AI adoption and the benefits it will bring.

Juras Juršėnas, Chief Operations Officer at Oxylabs, said, “Many industries have been hesitant to adopt AI tools, doubting the impact they provide and being unwilling to risk the investment. Yet, AI tools are perfectly suited for larger-scale web data collection. Building and maintaining data pipelines involves lots of moving cogs and an abundance of repetitive tasks that AI can streamline.”

67% of respondents stated that AI will primarily help build data parsers across the web intelligence industry, followed by target unblocking at 59%. Nevertheless, 43% of respondents saw target unblocking as the most suitable for AI applications, as it is a more sophisticated task and a more fitting challenge for the newest technological developments, such as AI tools.

Julius Černiauskas, Chief Executive Officer at Oxylabs, said, “Parsing has remained time-consuming and challenging even when AI was applied, as there have been few exceptional AI-based solutions specialised in data parsing on the market up to now. However, new AI tools such as Oxylabs OxyCopilot are emerging to assist with this. These tools include custom parser builders which enable data parsers to be created in minutes, and request builders to help define scraping requests with only minimal parameters.”

The combination of these two functionalities will help new AI tools to save developers’ time, avoid business costs created by heavily delayed data collection, simplify onboarding for those with less web scraping experience, and reduce spending on internal computing resources.

“The web intelligence industry is a long way from full AI integration, but new innovations are making a major difference in streamlining many tasks. In 2025, web intelligence companies should closely watch how these trends are developing and take steps to leveraging new tools at every opportunity”, concluded Černiauskas.

Complexity, scalability and compatibility remain challenging - 70% data workers struggle with pipeline management.

Cybersecurity strategies are failing

Posted 4 days ago by Phil Alsop
Cyber firm pleads with enterprises to wake up to the data security crisis before financial and legal fallout becomes catastrophic.
Avanade is unveiling the Avanade Intelligent Garden at this year's RHS Chelsea Flower Show in celebration of its 25th anniversary.

AI agents break cover

Posted 1 week ago by Phil Alsop
In a global survey of IT leaders, Cloudera found that enterprises are keen on AI agents, but fears around data privacy, integration, and data quality...
Economist Impact is pleased to announce the inaugural AI Compute summit, scheduled for May 22nd 2025, at the Scandic Copenhagen in Copenhagen. This...

Majority of AI projects don't make it to market

Posted 1 week ago by Phil Alsop
SS&C Technologies Holdings has published findings from a new survey: governance, process orchestration and strategic planning are critical to...

Security and compliance risks make VPNs obsolete

Posted 1 week ago by Phil Alsop
Zscaler has published the Zscaler ThreatLabz 2025 VPN Risk Report, commissioned by Cybersecurity Insiders, which highlights the widespread security,...

AI tops tech growth charts

Posted 1 week ago by Phil Alsop
Despite high interest rates, economic slowdown, stricter regulations on big tech and AI, Trump's tariff policies, and global trade wars, which hit...