Continued growth in AI and automation for data collection

The web intelligence industry has decisively turned to artificial intelligence as the main method to improve its operations.

  • Monday, 20th January 2025 Posted 2 months ago in by Phil Alsop

77% of web intelligence professionals have tried using AI-powered web scraping tools, with 86% stating that it saves time and resources. This is according to new research from leading web intelligence acquisition provider Oxylabs.

In cooperation with Censuswide, Oxylabs surveyed 506 software engineers and web scraping professionals from UK and US companies, gathering insights on the issues faced when collecting web intelligence and the impact it brings to businesses.

Findings from the research confirm that with advances in AI model development, the demand for high-quality public web data is accelerating leading into 2025. 47% of respondents claim to use AI-powered solutions regularly, while 20% have not used them so far but are keen to do so, demonstrating that the web intelligence industry has embraced AI adoption and the benefits it will bring.

Juras Juršėnas, Chief Operations Officer at Oxylabs, said, “Many industries have been hesitant to adopt AI tools, doubting the impact they provide and being unwilling to risk the investment. Yet, AI tools are perfectly suited for larger-scale web data collection. Building and maintaining data pipelines involves lots of moving cogs and an abundance of repetitive tasks that AI can streamline.”

67% of respondents stated that AI will primarily help build data parsers across the web intelligence industry, followed by target unblocking at 59%. Nevertheless, 43% of respondents saw target unblocking as the most suitable for AI applications, as it is a more sophisticated task and a more fitting challenge for the newest technological developments, such as AI tools.

Julius Černiauskas, Chief Executive Officer at Oxylabs, said, “Parsing has remained time-consuming and challenging even when AI was applied, as there have been few exceptional AI-based solutions specialised in data parsing on the market up to now. However, new AI tools such as Oxylabs OxyCopilot are emerging to assist with this. These tools include custom parser builders which enable data parsers to be created in minutes, and request builders to help define scraping requests with only minimal parameters.”

The combination of these two functionalities will help new AI tools to save developers’ time, avoid business costs created by heavily delayed data collection, simplify onboarding for those with less web scraping experience, and reduce spending on internal computing resources.

“The web intelligence industry is a long way from full AI integration, but new innovations are making a major difference in streamlining many tasks. In 2025, web intelligence companies should closely watch how these trends are developing and take steps to leveraging new tools at every opportunity”, concluded Černiauskas.

Failure to prioritise testing and integrate generative AI tools raises concerns as agentic AI adds pressure.

CIOs 'overspend' on cloud

Posted 4 days ago by Phil Alsop
43% of CIOs say their CEOs and/or board of directors have concerns about their company’s cloud spend.
Research revealed at Coterie Connect event highlights shifting team structures, evolving skills priorities, and urgent training needed for partner...
Endava has launched its latest research report “AI and the Digital Shift: Reinventing the Business Landscape”.

3,000% surge in enterprise use of AI/ML tools

Posted 1 week ago by Phil Alsop
Zscaler has released the ThreatLabz 2025 AI Security Report, based on insights from more than 536 billion AI transactions processed between February...
Over one in four (28%) British small business owners have used AI tools to help run their business.

Tech fragmentation cited as biggest cyber challenge

Posted 1 week ago by Phil Alsop
New Palo Alto Networks data shows 82% of UK organisations confident in their use of AI, despite AI being identified as biggest cyber risk for 2025.
MIT researchers crafted a new approach that could allow anyone to run operations on encrypted data without decrypting it first.