Big data and analytics?

The Big data and analytics question is at the forefront of many business leaders thinking but it’s worth taking a step back and looking at some of the wider macro-economic trends. Since the post war era, UK productivity has risen steadily when averaged over the decades but dropped significantly following the 2008 financial crisis and is now only slowly reaching post crisis levels. In fact, the Office for National Statistics has stated that in the last quarter of 2017, productivity of British workers increased at the fastest rate in more than six years. By Pinakin Patel, Senior Director Solutions Engineering at MapR.

Monday, 26th November 2018 Posted 6 years ago in by Phil Alsop

There are still two large challenges organisations face to positively impact their business outcome when it comes to data and analytics. The first is the completeness of the data to give an un-biased context of a business moment and this leads to the timeliness of the resulting action. The extent to which these challenges are addressed depends on the maturity of a data driven business which includes the overall data culture and data literacy across an organisation.

Innovative data driven-organisations are asset light businesses-like Uber and Airbnb and have used data to significantly challenge the status quo and transformed industries overnight.

Data is essentially just a resource that without analytics to focus it around solving a problem is effectively an untapped potential. Hence, re-imagining business activities in terms of the data that can be collected both internally and externally and how the data can help understand the activity and used to highlight or predict a business outcome is the first step.

Human capacity to understand the data and the analytics is the second stage. This can be thought of as: what action and when will have a direct impact on the business outcome that can be significantly higher than the activity without data and analytics. For example, a large semiconductor manufacturer leverages sensors across their manufacturing process to improve chip yield percentages. In this example, they analyse heat and vibration and other machine sensor data including thickness uniformity; electrical properties and other stats. As a result, they can identify and address product flaws much earlier in the manufacturing process and this has resulted in tens of millions of dollars in savings and increase yield of their final product.

There is still a shortage of data scientists and some management reluctance to sign-off on big data projects where the benefits are still to be determined. This is changing, rapidly in the case of certain innovators like HP who instigated major big data led digital transformation that have helped generate insights that lead to raised efficiency, productivity and other tangible business benefits.

Productivity, insightful decision making, and forward planning can all be improved with a good understanding of the underlying trends that impact each business and the wider market. Data driven market forecasting for example has been around since the 15^th century. Early economists such as Sir Thomas Mun, an economist and director of the East India Company, used trade data to help it to eventually become the largest private company of its era. Mun and other early data scientists mixed both raw numbers with a deep knowledge of the market, human nature and likely external factors such as technology innovations, politics, and as a sea-based trading company, even the weather, to create corporate strategy and business execution processes. This same mix of skills, knowledge and data are available today and organisations need both the will and often courage to capitalise on opportunities. HR departments have a major role to play in supporting these types of transformative strategies through recruitment, training and ongoing support. However, HR needs to be candid with senior executives about the viability of retaining the right staff internally and, in some situations, it might be better to work with external specialists such as contractors or solution providers.

Digital transformation has given business the ability to understand their business activities at a much more granular level and at scale than ever before. The resulting data-driven business needs a supporting data culture and data literacy which HR departments have a role in helping to establish and foster. This brings to light another key HR requirement which is helping to create collaborative environments that teams need to deliver these projects while meeting company polices and regulatory requirements.

Data analytics without a focus on business outcome can result in a resource-sapping distraction. To avoid this issue requires a combination of data experts and technologies that are necessary to uncover hidden patterns and un-known correlations. However, many businesses do not have the luxury of being born in a digital world. Hence, they have legacy of manual and automated processes; lots of business applications and as a result lots of data silos. In addition, the extent to which data has been digitised and readily available varies greatly.

As a result, organisations are initially looking at creating data fabrics that place all this structured and unstructured data along with free-flowing and historic data all in the same platform to allow analytics project simpler access to the complete view and not just a single slice. These data fabrics are based on open standards to make analytics tasks easier to handle but all still secured and managed via central policies so that organisations can meet data and privacy regulations.

With the complexity of a business environment, and as they are faced with deluge of data, emerging technologies in machine learning and AI are also key in helping businesses augment their human intelligence with machine intelligence in making sense of the complex but informative hidden patterns and correlations that can be found in the data at scale.