AI, automation, and web scraping set to disrupt the digital world in 2025

The development of artificial intelligence (AI) depends on public web data, which is used in large volumes for training algorithms. Experts from Oxylabs, a global leader in creating solutions for gathering such web intelligence, discussed their predictions for major AI and machine learning (ML) developments in 2025 in their industry and other spheres.

The company’s CEO, Julius Černiauskas, said, “Web scraping professionals are generally happy with the results of AI adoption. Thus, we might see a proliferation in AI and ML-based web scraping solutions for target unblocking, proxy management, parsing, and other tasks.”

“I believe the usage of CDP browser tools will grow in 2025. These tools allow scraping with real-like browsers not easily detectable by the ever-improving anti-bot systems. Quite a few companies use AI to automate these browsers, and their number will probably grow,” specifies Černiauskas.

Tomas Montvilas, chief commercial officer at Oxylabs, said, “We will see a proliferation of AI agents and multi-agent systems as companies aim to develop AI architectures that can perform tasks autonomously or semi-autonomously. Automation of tasks performed on the web will be a big focus next year, leading to a growing need for browser management and web crawling technologies.”

“Nevertheless, building agentic architectures in-house, without external help, will be challenging, as McKinsey predicts that three out of four such attempts will fail,” he added, emphasizing the importance of finding reliable partners in tech.

Juras Juršėnas, Oxylabs’ chief operations officer, agrees that AI tools will be increasingly crucial for IT professionals, “A new generation of AI-assisted tools for developers will go mainstream. This will affect many industries, including web scraping, where we are already introducing such tools this year and certainly not planning to stop moving in that direction.”

“Another area where AI usage will be accelerating rapidly is the cybersecurity cat-and-mouse game. There will be a lot more cases where AI is used for both fraud and countering it. AI-powered web scraping technology will also play a role here, helping cybersec professionals identify risks faster and on a larger scale.”

As with any other industry related to data and AI, web intelligence gathering will be affected by the developments in AI regulation. Denas Grybauskas, head of legal at Oxylabs, watches these developments closely.

According to Grybauskas, “As AI regulation matures and the first comprehensive AI law, the EU’s AI Act, comes into power, we will see how different regulatory frameworks affect the development of AI and ML solutions in business and other areas. Lessons learned from practice might inform future lawmaking in the EU and jurisdictions that are yet to enact AI laws of similar scope. This year, we might also see how AI governing bodies throughout the globe interpret AI laws and the kind of decisions we might expect in the future.”

“Additionally, the improving AI algorithms enable hyper-personalization of online offers and ads, possibly tailoring them to the unique specifications of individual users. These growing powers of algorithms raise privacy concerns that lawmakers will have to look into, possibly leading to new developments in how data and AI are regulated.”

Considering broader AI implications on the market, Juršėnas said, “I expect local generative AI (Gen AI), like Apple Intelligence, that runs on PCs and cell phones to leave a growing mark on everyday user experience. Furthermore, similarly to how image generation models boomed a few years ago, we might see video generation models booming in 2025.”

Meanwhile, Montvilas points to the workforce dynamics, “The evolution of the workforce will continue, including an improved division of labor between humans and AI, as companies gain some insight into which tasks and to what extent can be delegated to AI for the most benefit. In our industry, web scraping tool developers will certainly utilize these insights to apply AI where it can add the most value.”