The terabytes and petabytes are mounting up but how successful have Big Data initiatives in the UK been so far?

Big data is one of the most important drivers of digital transformation, but data alone is no guarantee for success. Many UK companies are still finding their way around Big Data, testing technologies, use cases and tools, as well as new operational models.

Friday, 4th May 2018 Posted 6 years ago in by Phil Alsop

But what are the biggest business and technology challenges companies face with Big Data? How successful have initiatives been so far? What is the best approach to implementation, and what are the solutions? WHISHWORKS has conducted a survey among Big Data strategists, architects and users and has consolidated the findings in its latest report: The State of Big Data in the UK 2017/2018.

Implementation and success

According to the survey, only 18 percent of companies in the UK have fully implemented Big Data. Many companies are still at the ‘experimentation stage’, as evidenced by 76 percent that are actively investigating (29 percent) or have tentatively started (47 percent) Big Data initiatives. The sectors leading the way are Media and Entertainment (with over 75 percent of the data being processed by a Big Data solution), followed by Telecommunications and Financial Services, which have both been using Big Data successfully for over three years.

70 percent of the companies that have started implementing Big Data, said their initiatives have been ‘somewhat successful’ so far, while another 30 percent deems their projects to be ‘very successful’ – encouragingly, none of them have failed or been unsuccessful.

Challenges

IDC has recently predicted that the big data technology and services market is expected to reach $58.9 billion in 2020, whereas Burning Glass, a jobs analytics firm, found that big data skills saw the biggest increase in demand (3977%) between 2011 and 2017. With such record growth, a surge in demand for data processing and analysis talent should be expected. This is evident in the results of the survey, where lack of skillset was cited as the biggest challenge to businesses in implementing Big Data (71 percent).

Cost concerns (53 percent) and data integration issues (47 percent) were also high on the list of challenges identified by the respondents.

On the business side, the majority of respondents (65 percent) highlighted limited understanding across the organisation as the number one challenge they are facing.

Expected Impact and Value

Companies are using Big Data for projects relating to Artificial Intelligence and Machine Learning (53 percent), followed by Digital Technologies such as mobile and social media (20 percent), and Cloud Computing and Internet of Things both with 14 percent. “Whereas Digital Technologies, Cloud computing and the Internet of Things bring more technology-related benefits, Artificial Intelligence and Machine Learning have a more direct impact to the business and it is therefore no surprise that the majority of organisations expect that these two fields will have the greatest impact,” commented Vijay Bhoomireddy, Big Data Architect at WHISHWORKS.

The survey also shows that the business areas where Big Data is expected to bring the most value, are product marketing (55 percent), revenue from current activities (54 percent), process optimisation (54 percent) and business transformation (47 percent).

Jumping through Hadoop

Apache™ Hadoop® is the open-source software framework to deal with Big Data, and is a good place to start experimenting and understanding Big Data. As projects mature however, commercial platforms like Hortonworks, MapR Technologies and Cloudera are the preferred Hadoop-based solutions, with 75 percent of businesses surveyed employing or planning to deploy one of them for their Big Data projects.

Business leaders and IT managers should review the corporate objectives and commercial requirements, as well as the technology available, to devise a strategy to implement the correct solution for digital transformation.

What next?

“As companies gain more understanding of how they can benefit from Big Data, the rate of adoption is going to increase exponentially. In 2018, we expect to see companies to process more of their data with Big Data technologies and start implementing more complex and/or impactful use cases,” commented Edward Davies, Sales Director at WHISHWORKS. “It is also going to be interesting to see how different industries will evolve taking advantage of the new opportunities Big Data presents, and what new ‘disruptors’ will surface.”