On 19th July 2024, services and industries around the world ground to a halt. The cause? A faulty automated software update issued by CrowdStrike. While widely known by security experts, the sheer impact of such an update was made painfully clear to the average person, affecting countless businesses and organisations in every sector, including automotive manufacturers.
An example of this was seen with Tesla – the automotive giant had to shut down assembly lines in Texas and Nevada because of the incident. Computers, servers, and machinery were impacted, and employees were sent home. Despite the swift identification of the issue and quick fix for the problem, it was the impact and recovery process on the customers’ environments that were disastrous and lengthy. This serves to highlight the risk that automotive manufacturing can face without proper observability in place.
Beyond major outages, production lines still face challenging delays and performance issues that are hard to pinpoint and lead to poor user experiences. These unexpected delays and downtime result in higher costs, reduced output, and longer waiting times for customers, potentially hurting brand loyalty. Getting systems back online quickly is crucial for manufacturers, as just one minute of downtime can cost organisations tens of thousands of pounds.
The challenge of point tools
Traditional observability data from point tools – specialised solutions that monitor a single area, such as cloud infrastructure – often makes it hard for IT teams to identify the root cause of issues. Relying on these monitoring tools creates a significant blind spot because they lack the scope and depth to reveal the truth behind, say, why there was a latency spike in robotic assembly lines or why factory-floor device performance drops after a failed software update.
Blind spots exist because these tools fail to map the complex communication paths that connect the factory floor to critical applications hosted in the cloud, at co-location sites, or by Software as a Service (SaaS) vendors. To address this gap, manufacturers need enterprise-wide observability, from the factories to their private data centres and the public cloud, wherever their application services are hosted.
Since point tools only rule out a portion of the network as the source of the problem, they fall short in providing comprehensive observability. Packet data complements these tools by seeing the entire communication path. It provides a complete, ecosystem-wide picture of the communications for rapid, true root cause analysis.
Packet-level data: the ground truth for network observability
The definitive source of truth for understanding network activity is packet data. Deep packet inspection leverages the raw evidence from network activity to provide actionable insights. In real-time, it can show whether there's a problem in an automated assembly line, identify the cause of a network outage, or pinpoint the source of a problem to an application server hosted in the public cloud.
For automotive manufacturers, real-time visibility into network activity, whether on-site or remote, is essential for staying on schedule, maximising assembly line uptime, and detecting when faulty equipment may be a bottleneck in the communications path.
For example, an assembly line could be running smoothly, then a backup begins when a robotic arm experiences a half-second latency spike, causing a micro-stoppage or disrupting synchronisation with other robotic arms. If this occurs once, the impact on manufacturing might be minimal; but if it happens repeatedly across multiple vehicles, the factory will fall behind.
However, with packet data, the IT team can quickly determine if the delay is caused by increased network traffic or a DNS failure. This deep visibility can quickly pinpoint the ‘why’ and the ‘where’. This is especially important for remote sites that often don’t have dedicated on-site IT staff. Instead of sending a specialist out to troubleshoot, these insights help a centralised team investigate the issue remotely, saving valuable time and money.
With smarter factories, over-the-air software updates are becoming more commonplace, whether to patch up a security threat or implement newer software for factory-floor devices. Unfortunately, as we saw with the CrowdStrike incident in 2024, these types of routine updates can lead to performance issues, and the underlying reasons for these issues may not be immediately clear. This is where packet data comes in, by providing network context to identify exactly where and what caused the problem in the first place, significantly reducing the mean time to repair (MTTR) and minimising the impact of any IT incident.
Crucially, because as much as 90 percent of MTTR is spent identifying and understanding the root cause of a problem, improving this diagnostic phase – known as mean time to knowledge (MTTK) – is key to accelerating recovery and restoring critical services with minimal disruption.
With the cost of downtime measured in tens of thousands of pounds per minute, addressing network blind spots with deep packet data has become a business imperative. This isn’t just about meeting production goals. It’s about building a resilient, intelligent manufacturing operation ready for the future.
