In the wake of the recent Veeam On tour, I sat down with Veeam's Tim Pfaelzer, the GM and senior vice president for Veeam EMEA and Mena Migallly, the regional vice president for Veeam EMEA East with one question burning a hole in my brain: Is AI really making the world more dangerous?

This is in lieu of researcher Jacob Coxon's dramatic departure from Anthropic last month, after sending shockwaves through the tech industry when he publically accused AI giant Anthropic of prioritising profitability over human safety.

Pfaelzer explained that while artificial intelligence may not be making the world inherently riskier, it is changing the shape of risk.

He noted that every major wave of technology has triggered anxiety, citing the arrival of personal computers, which those in the early wave similarly feared that machines would take over from people. And then there was Y2K, where all banking infrastructure was surely doomed to collapse at the stroke of midnight at the turn of the millenium, like the fabled princess's chariot that suddenly reverted back into a pumpkin. Neither fear proved to be true.

But, when it comes to AI, there is still genuine cause for concern.

''My personal view is that there are now more sources that can create risk, and it's moving faster. This means we all just have to adapt to things and how they happen. Even more importantly, people now need to be prepared for new ways of working,'' Pfaelzer elaborated.

He further drew an important distinction between the total amount of risk and the number of ways it can emerge.

Tim Pfaelzer, the GM and senior vice president for Veeam EMEA

AI’s Achilles heel

The problem with AI is that it is a doubled-edged sword, simultaneously creating  new opportunities to work faster and more intelligently, while also giving bad actors and cybercriminals more options to launch their own attacks.

So really what AI is bringing us is not simply “more risk”, but more varied risk moving at far greater speed.

That acceleration is already eroding the margin for human intervention. Pfaelzer highlighted an example in which data, including backups, disappeared within nine seconds. While traditional incident-response strategies were designed around the assumption that admins and management would have time to respond to a breach, in a nine-second window, that assumption, and all the already established strategies, collapse.

Mena Migally added a second dimension: the attack surface. This, he admitted, has expanded dramatically. 

The reason, he explained, is AI-enabled environments now connect more applications, agents, systems and data sources at once. ''Security teams used to focus on looking for vulnerabilities in one system or in one environment, but now everything is open to potential attack at the same time,'' he explained.

Rogue Agent.... or Something Else

The same tension appears in agentic AI. Because there have been stories in the wild about agentic AI going rogue, (recalling Agent Smith from the Matrix and conjuring fears that that movie may have been more prophetic than fictional) and exceeding its parameters. That seems to point towards a concern that agentic AI isn't contained by guardrails and can potentially run amok.

But Migally argues that the more common problem is an agent doing exactly what it was instructed to do, but having far more access than anyone intended.

He explained that an overprivileged agent can retrieve or alter information across systems, not because it is malicious, but because its permissions were poorly defined. The governance questions that this raises are therefore practical ones: what can the agent access, and what is it allowed to do with that access? And, the most important question is whether that is a human's responsibility to define more clearly.

Pfaelzer likened the issue to autonomous driving. While an autonomous vehicle may fail to brake without any malicious intent; the failure still has consequences. This means that the human operator cannot simply stop paying attention. He pointed out that in the same fashion, useful AI systems still require active supervision.

The challenge becomes even greater at scale. He pointed out that one employee may be supported by dozens of autonomous agents, multiplying the number of digital actors that must be trained, monitored and governed.

Mena Migally, regional vice president for Veeam EMEA East

Who cut the brakes?

My next question to both was whether organisations are moving too quickly to adopt AI, and perhaps being more driven than FOMO (Fear Of Missing Out) than by wisdom.

For Migally, the rush to adopt AI is understandable. He noted that the benefits are immediate and attractive: operational efficiency, access to expertise and the ability to close skills gaps. That can encourage organisations to deploy first and ask difficult control questions later. However, he noted that he is beginning to see companies apply the brakes, conduct due diligence earlier, restrict agents’ access and pursue productivity gains without accepting uncontrolled exposure.

Both leaders reject the idea that AI enables a “set it and forget it” future. Pfaelzer compared today’s agents with the machines of the industrial era, pointing out that a production line left unsupervised overnight might manufacture a thousand faulty components.

AI, however, can make the equivalent mistake in seconds. What I found particularly interesting from Tim’s presentation and then our subsequent interview is that for all that AI is disrupting, one thing that hasn’t changed is the need for (human) oversight.

This, perhaps, is even more important, because of the fact that AI is an accelerant, accelerating not only the pace of innovation, but also the potential speed and scale of failure and making major missteps.

The inescapable human factor

The crux of my interview with both Veeam execs is that while AI is here, and does bring both upsides and downsides, human responsibility remains at the core of the AI conundrum.

Perhaps it is people, and human error, that are the greater risk, than a rogue AI agent, because one strongly made point by both was that it is up to organisations to choose, install, train and give AI systems and agents permission.

In a nutshell, poorly governed agents are therefore symptoms of poor human decisions rather than replacements for human error.

Advice for any organisation

Pfaelzer took that perspective further, arguing that organisations must remain deliberately suspicious, (and perhaps adopt that zero trust attitude that is so common in the cybersecurity world), continually asking whether an agent is properly trained, whether it has excessive privileges and whether its output matches its intended purpose.

For Migally, the real threat in an AI era is another human one, that of complacensy.  He stressed  that trust should be earned through checking, testing and continued oversight, whether the subject is a person or an AI agent. The objective is not to reject automation, but to establish controls strong enough to make its benefits sustainable.

At this point, organisations may have their own question burning a hole in their collective brains: What should we do with AI?

Pfaelzer and Mena suggested a two part strategy. The first entails reducing the likelihood of failure through visibility, least-privilege access and governance across their data estate. At the same time, organisations must accept that no degree of control can guarantee perfect implementation and prepare to recover rapidly when something goes wrong.

Is this the end, or just the beginning?

In the agentic era, resilience is no longer a fallback that organisations rely on after something goes wrong. Rather, it should be a capacity that is already in place before it is needed.

To my mind, this kind of resilience is similar to adaptability, another quality that was often touted during the 2010s when everyone and their dog had the word innovation on their lips.

According to the execs,  the organisations best positioned to benefit from AI will be those that match AI’s speed with equally fast detection and recovery, and its scale with disciplined human supervision.

So, will AI bring about the end of humanity or lead to an extinction level event?

Tim's callback to and comparison of the AI era with the onset of the PC was a relevant one, not just as a reassuring example that AI probably wont spell the end of the world. Human stupidity, or carelessness, however, might. 

It reminded me of Apple TV's Dark Matter series, whose parallel-world hopping protagonists find themselves in a world where humanity never went to the moon, and the invention of the silicon chip, and thus the onset of the PCs and Internet, never happened. That world depicts a similar dualistic, double edged sword nature as AI is heralding, where life is both better in some ways (more human to human connection) and worse in others (much less access to information and innovation).

Whatever world the invention of AI leads us towards, I think heeding Pfaelzer and Migallly’s advice is valuable, to ensure that the world we end up with is one we are willing to stay in.