Top trends for modernising the data warehouse

WhereScape has published the results of a global survey by the Business Application Research Center (BARC), commissioned by WhereScape. The survey revealed that nearly half of respondents found time-consuming development processes at the top of the list of primary drivers for data warehouse modernization as companies look to keep pace with increasing complexity in the data landscape. Lack of agility in infrastructure and internal processes, as well as a lack of business and technical know-how, however, are limiting business’ ability to act.  

“This research gives statistical evidence and puts the BARC stamp on concepts that WhereScape has been evangelizing for some time. In order for companies to succeed today, we must automate repetitive manual tasks that force too many developers to solve puzzles and fight fires arising from outdated tools and methodologies. Only then can companies focus forward on innovation and agile collaboration to supply reliable data to the business at the right time,” said Rob Mellor, vice president and general manager EMEA, WhereScape.

Drivers for modernization

44% of survey participants found that time-consuming development processes were the most daunting tasks in the data warehouse landscape and the primary reason to modernize.  This response was followed by restricted support of self-service BI (35%) and lack of ability to provide required data (30%). Of note, the inability to meet compliance and regulatory requirements was only seen as a major challenge by 14% of participants. Even less (11%) found insufficient technology or lack of functionality to be a limiting factor, suggesting technology has reached a sound level of maturity.

Methods of modernization

Data warehouse and ETL automation is the most important approach to modernizing the data warehouse (60%). Correspondingly, data integration and data warehouse automation are the technologies most frequently utilized. By trend, however, enterprises with more mature data warehouse landscapes are more geared towards moving into the cloud, and report less infrastructure and process challenges - clearly profiting from investments in a flexible data and process architecture and agile data services.

Challenges faced in modernizing the data warehouse

Participants cited lack of agility in infrastructure and internal processes (39%) as well as lack of business and technical know-how (38%), as the key factors limiting the ability to move forward with data warehouse modernization. Data and analytics literacy is not only an immensely important success factor for building an effective, cost-efficient and sustainable data warehouse environment. These skills are invaluable for the transformation to the digital enterprise, yet they are rare. Enterprises with less advanced data warehouse environments struggle more than those who are more advanced. However, it is vital to realize that this area requires investment in existing employees in any case.

Leaders, followers and laggards

In order to characterize the status quo of survey participants’ current data warehouse and analytics landscape, three groups of respondents were created: leaders, followers and laggards (see definitions below). Here were the standout statistics within the groups surveyed:

-       The data warehouse/ETL automation approach to modernization was a popular approach with nearly half or more of respondents from all groups (leaders, followers, and laggards).

-       For leaders, real-time processing of data is the most important (41%). This is due to real-time processing being one of the most important prerequisites for decision automation and being an important foundation for agile, interoperable data services.

-       For leaders, analytical databases are utilized much more intensively than by followers and laggards (41% compared to 27% overall).

-       For leaders, cloud services play an important role in approaches to modernization of the data warehouse with 46% citing cloud services as a promising alternative with regard to resource and cost efficiency of hardware.

-       Laggards in particular suffer from a lack of support from executives for data warehouse modernization initiatives (44%).

Concluded Mellor: “In order for IT to advance and deliver timely business insights needed in today’s era of exploding data, an automation-first approach to data warehouse modernization is essential. Leading organizations are recognizing the need to focus skilled development resources away from repetitive coding and onto higher value projects by arming developers with the automated tools necessary to agilely deliver.”