“Unknown unknowns” holding back Hadoop adoption despite insight benefits

Half of organisations believe that the most significant benefit of Hadoop as a data integration platform is its ability to provide them with insights they have never had before – but a fundamental lack of understanding of the technology, and concerns over its maturity, are holding back widespread adoption. It is time for more vendors to get involved in the open source community and contribute to both making Hadoop more functional and educating organisations as to its benefits to them, according to Nejde Manuelian, Director of EMEA Operations at Syncsort.

50 per cent of organisations questioned as part of Syncsort’s EMEA Big Data and Hadoop Trends survey[1] cited “getting insights you never had before” as the top benefit of doing data integration in Hadoop, ahead of “reducing overall data costs”, “enabling more complex data processing”, “making storage more economical”, “making storage and compute” more scalable”, and “increasingly agility and flexibility”.

When it came to describing the barriers holding them back from adopting Hadoop, almost half of organisations (49 per cent) cited “uncertainty or a lack of understanding” as their top concern. Other factors included “connectivity and integration with existing applications”, “the need to learn new skills”, “the difficulty of finding programmers”, and “Hadoop being more expensive than anticipated”.

Manuelian believes that such concerns have merit, and that Big Data vendors will need to do more to help the open source community to develop Hadoop if organisations are to take the maximum benefit from the platform when their traditional data handling tools cease to be sufficient:
“It is still early days, but Hadoop can provide some serious benefits for organisations – including bringing greater flexibility and scalability to their data handling and, crucially, making it less complex and expensive. Our research shows that organisations are already aware of these benefits. It is the ‘unknown unknowns’ of Hadoop that are holding it back from being deployed more widely in the enterprise setting,” he said.

“Many organisations are saying that they don’t know what they don’t know about Hadoop,” Manuelian continued. “And it is this uncertainty that is producing most of the other concerns they are mentioning. Hadoop is not yet the mature data integration solution that the enterprise will need in a future shaped by Big Data – but rather than adopting a wait-and-see approach, more vendors should be following the lead of early movers who have already contributed enhancements to Hadoop’s enterprise capabilities.”

Manuelian believes that it is in the best interests of vendors and end users alike if the industry comes together to accelerate the maturation of Hadoop: “The benefits of Hadoop – and the need for an alternative to current tools – are clear. Putting aside the usual selfish concerns about competition at this early stage in the interest of collaborative development of the platform will be crucial if vendors and end users alike are to reap those benefits in the long run,” he concluded.