Paul Scharre on how AI could transform the nature of war

Transcript

Cold open [00:00:00]

Paul Scharre: How do you end a war that’s happening at superhuman speeds? If our competitors go to Terminators and their decisions are bad but they’re faster, how would we respond? There’s this incentive towards faster reaction times and decision making. They have to go faster to keep up.

I think we have a really interesting example in financial markets, stock trading, where humans can’t possibly intervene in milliseconds, and then we’ve seen examples like flash crashes. Could we have something like a flash war, where interactions are so fast that they escalate in ways that humans really struggle to control?

The ugly reality is likely to be that, politically, people will have to suffer and die for wars to end.

Who’s Paul Scharre? [00:00:46]

Luisa Rodriguez: Today I’m speaking with Paul Scharre. Paul is a former US Army Ranger who served in Iraq and Afghanistan, the current vice president and director of studies at the Center for a New American Security, and the award-winning author of two books, Army of None: Autonomous Weapons and the Future of War and Four Battlegrounds: Power in the Age of Artificial Intelligence. He also worked at the Pentagon where he led the team that wrote the US military’s first policy on autonomous weapons.

Thanks for coming on the podcast, Paul.

Paul Scharre: Thanks for having me. Very excited for the discussion.

How will AI and automation transform the nature of war? [00:01:17]

Luisa Rodriguez: You expect AI and automation to transform the nature of war. Can you talk concretely about what that will look like?

Paul Scharre: I think we’re already starting to see artificial intelligence play an important role on modern battlefields. I think over time, we’re going to see AI take on more of the cognitive dimensions of warfare, and get to a place where — and this might take several decades — the speed and tempo of war could start to really push at the boundaries, maybe even exceed them, of what humans can do.

So you can envision a world in the future where you get this tipping point, what some Chinese scholars have called a “battlefield singularity”: an idea that the speed and tempo of war outpaces human control, and war at a large scale shifts to really a domain of machines, and machines making decisions.

Luisa Rodriguez: Yeah, the idea of a battlefield singularity is extremely interesting to me. But I want to step back briefly and just try to understand how exactly AI will be integrated into weapons systems, and then how it will affect how wars are fought and won.

If I understand correctly, autonomous weapons systems are complete weapons platforms that, once activated, can select and engage targets without further human intervention. Can you give a couple examples of the most advanced autonomous weapons systems that are being developed today?

Paul Scharre: Sure. Conceptually, an autonomous weapon is really one where the weapon itself is making its own decisions on the battlefield about who to kill.

I’ll make an analogy first to cars. Conceptually, the idea of a self-driving car is pretty straightforward. You can imagine cars that don’t even have a steering wheel. Certainly people have designed them where the car is totally driving itself. Now, in practice, as we see the technology evolving, it’s these sort of incremental movements: you have incremental advances in autonomy and automation in cars — like intelligent cruise control, automatic lane keeping, automatic braking, automatic self-parking — like Tesla, that has more incremental self-driving features, but you can see this path towards a completely self-driving car.

We’re seeing a very similar thing inside militaries: militaries are incrementally automating different tasks that are used in finding targets, in processing those targets, in presenting that information to decision makers, in missiles and drones that would be able to carry out attacks.

Luisa Rodriguez: Can you give a few examples of, as it advances, maybe quite significantly, what are longer-term visions for how AI and autonomy could make really big differences to how these weapons systems work? I’m sure there are loads of different places where these things will be integrated. I’m most curious about the places that could most drastically change how wars are fought and won.

Paul Scharre: I think one major paradigm shift that could occur, and is probably eventually likely to occur over the next several decades, is towards swarming warfare. Where you can imagine large numbers of autonomous drones in the air, at sea, undersea, on land, that are networked, that are working together cooperatively and autonomously adapting their behaviour on the battlefield to respond to events.

Right now we’re seeing massive numbers of drones deployed in Ukraine, certainly tens of thousands of drones on the front lines. But those drones are not only remotely controlled for the most part; they’re not really working cooperatively in any way. So even if humans had the ability autonomously for the drone to go out and find its own target, having 10,000 drones that are independently finding targets is very different than 10,000 drones that are working cooperatively together.

You could have much more dramatic effects in the battlefield by having swarms that are able to simultaneously attack from multiple directions, have self-healing communications networks, self-healing minefields; the ability to react to what humans are doing, to what the enemy is doing in real time — and at a faster not only speed but also scale of coordination than is possible with humans.

And I think the real dramatic change here is not actually in the physical technology. I mean, drones are interesting, they could do neat things, but it’s in this sort of cognitive dimension — and in particular here of what the military would call “command and control.”

Militaries today are organised in this very hierarchical fashion: you have teams and squads and platoons and companies and battalions, and you have these organised predominantly because of the limitations of human cognition. So if you put a human commander in charge of 10,000 soldiers, and they were directly issuing orders to each 10,000, that would be totally impractical. There’s no way to do that. That’s not how militaries are organised, that’s not how corporations are organised.

If you look at sports, it’s really interesting that a lot of team sports have somewhere between maybe five to a dozen or so players on the field. Now, imagine a game of soccer where you had 100 players on each side and 50 balls: you’d have to have a completely different way of organising that.

But robots or swarms could do that differently. They could perfectly coordinate their behaviour and ensure that they’re optimally using those resources to hit the soccer balls, go up to the enemy targets, whatever it is. So I think that’s a potentially really dramatic shift in how militaries fight in the future.

There is this vision of a possible future that, as militaries integrate artificial intelligence and autonomy more fully across the force, we might reach some tipping point where the pace of combat action is just too fast for humans to respond, and humans have to be completely out of the loop.

I think what’s scary about that possible vision is that humans are then no longer in control of violence and warfare. And that raises moral questions, but it also raises just really fundamental questions of how do you control escalation in wartime? How do you end a war that’s happening at superhuman speeds? And we don’t have good answers for that. I think maintaining human control over warfare is absolutely essential to making sure that we can navigate this transition towards more powerful AI in a safe way.

Luisa Rodriguez: Just to make sure we get a concrete picture of what this battlefield singularity, or sometimes called hyperwar, would look like: Can you describe what it looks and feels like? What kinds of weapons? How have they been automated? What do conflict engagements look like? Are there any humans in the loop at any level?

Paul Scharre: Let’s start with where we are today, and I want to kind of paint a picture for how that might grow over time. So since at least the 1980s, countries have had automated air and missile defence systems that can shoot down incoming threats when the speed of these incoming missiles or rockets or artillery aircraft are just too fast for humans to respond.

For example, a US Navy warship has an automated mode on the air and missile defence system that can be activated where there might be missiles coming in — and there’s just so little time for humans to respond, and you might have multiple threats coming from different directions — so that then the machine, once it’s activated by people, can automatically sense all these threats and shoot them down.

Now, we’ve had these systems around for decades. They haven’t really been widely used in conflicts in these automated modes. And there have been a couple examples of accidents: there were a couple fratricide incidents in 2003 with the Patriot air and missile defence system. But that’s something that we have some experience with: there is this very narrow domain today of machine control over warfare where really, humans just can’t be in the loop in this area.

I think what I would envision is that this domain of machine warfare grows over time, and then several decades from now, we end up in a world where something like that exists at a much larger scale along the entire front, where there are swarms of thousands of drones on both sides, and they’re dynamic and responding to enemy behaviour. And there are missiles being launched and striking targets, and there are AI systems identifying new targets which are moving and mobile. And humans can’t possibly be in the loop to respond to that enough: it’s too slow.

And humans are maybe observing this action. Maybe you could think of it the way a coach might on the sidelines, right? Humans could have some degree of direction of, “I’m going to change the higher-level guidance for these systems, or I might try to add new parameters to the operating systems.” But humans can’t really, in real time, intervene.

I think we have a really interesting example of this exact kind of behaviour in financial markets, stock trading — where there’s this whole new domain of high-frequency trading where humans can’t possibly intervene in the milliseconds that these algorithms are responding. And then we’ve seen examples of flash crashes that come with that. So I think the scary analogy there would be, could we have something like a flash war, where the interactions are so fast that they escalate in ways that humans really struggle to control?

I think that’s a really scary proposition. How do you find ways to stop that? In financial markets, they put in place circuit breakers that can take trading offline if we see movements that are too volatile. There’s no good way to do that in warfare. There’s no rough way to call timeout. How do you maintain human control over warfare that’s happening at superhuman speeds?

Why would militaries take humans out of the loop? [00:12:22]

Luisa Rodriguez: Can you talk about the incentives that lead to this kind of inevitable speedup and taking humans out of the loop? My sense is that currently the Defense Department and others who are going to be in charge of these decisions do not want to take humans out of the loop. So why does that seem like a likely thing that’s going to happen, and how does that drive things forward faster?

Paul Scharre: Yeah, that’s a great question. I think this push/pull is very common in these types of major revolutions in military affairs, where you have old institutions and ways of fighting that are not necessarily super enthusiastic about the new way of fighting. The cavalry, for example, wasn’t particularly enthusiastic about tanks.

And right now, certainly within the US military, there’s a strong belief that humans should remain “in the loop.” That’s not actually official US policy, but certainly when you hear US senior military officers talk about it, they’ll talk about it that way: that they want humans in control. I think because there’s just a healthy scepticism for all the reasons that everyone who’s ever interacted with AI could understand about these systems: that sometimes they get it wrong, and there’s value in humans making these decisions.

I think the ultimate arbiter is what works on the battlefield. That is what will drive how militaries change. Militaries tend to be often very conservative with these types of changes, in part because you never really know what’s going to work until militaries fight a war.

Luisa Rodriguez: Right. So the idea is there is this conservatism, and maybe it takes decades, but eventually the technology… I mean, it’s my suspicion that the technology just is very likely to improve enough that you’re really disadvantaging yourself if you don’t use it. Does that sound right to you?

Paul Scharre: I think that there’s a trajectory towards greater automation and greater speed and tempo of war. I do think that militaries have choices about exactly how they implement that technology. And the important thing for militaries — this is actually true of most military tactical revolutions — what matters most is not actually getting the technology first, or even having the best technology in some sense; it’s figuring out the best ways of using it. It’s figuring out, like, What do I do with a tank? What do I do with an aeroplane?

And I think there’s value to human cognition. There are lots of types of cognitive problems that at least today are very challenging for AI. And even if AI is cognitively better, there’s probably value in keeping humans in control of warfare. The question is how to maintain that balance in the best possible way. I think that that’s going to be a really important question in the next several decades.

Luisa Rodriguez: But do you think that value is really likely to persist long enough that, at some point at least one country decides that taking the human out of the loop is strategic and then does better? And if they do better, that creates this pressure for their adversaries to take them out of the loop?

Paul Scharre: So Former Deputy Secretary of Defense Bob Work, who was really a pioneer in bringing artificial intelligence into the US military, has this quote of, “If our competitors go to Terminators, and their decisions are bad, but they’re faster, how would we respond?”

Which is a colourful way for a senior leader to be talking about Terminators, but I think it does highlight this really difficult problem of this potential for an arms race in speed in militaries: that there’s this incentive towards faster reaction times and decision making that might pressure militaries to do the same — even if they don’t want to, if they’re not comfortable with that, they have to go faster to keep up, similar to what we’ve seen in financial markets with high-frequency trading.

And that could lead to a dangerous situation, where you have this dangerous arms race in speed in the military. I’ve heard some people argue we ought to have some limits on that. How do you put a speed limit on warfare? Seems like an appealing idea. I don’t know how you do that in practice to try to put brakes on this tendency, which is I think a big risk as militaries are adopting AI and autonomy.

Luisa Rodriguez: Yeah, interesting.

Paul Scharre: I think for autonomous weapons, the answer is going to be that there are strong competitive pressures to take humans out of the loop, at least in certain kinds of scenarios.

I would say, when I certainly talk to folks in the US Defense Department, that there’s still a lot of value that humans bring, and that these machines can make mistakes, so let’s not throw out human cognition. And even if we think our systems work great in testing, we don’t know what the enemy is going to do. We don’t know how our systems might interact with the enemy. And so particularly there’d be some battlefield applications where it might make sense to say that using autonomous weapons in this area is fine. I think so.

But certainly in controlling escalation, I think human control is really important. If you imagine something like the Cuban Missile Crisis: if we add a bunch of autonomy, I’m not sure that makes it safer. To me that makes the whole situation much more brittle and more likely to escalate in dangerous ways.

Because you can give humans really ambiguous guidance: you can tell humans things like, “You’re allowed to use force to defend yourself, but don’t start a war.” And a human can understand that. They might be like, “I don’t really know what that means in practice,” but we’re basically saying, “Look, I trust you, that you’re pretty smart and figure it out as best you can based on the situation that you’re in.” And humans can handle that.

I don’t know that AI systems, even as a consortium, will necessarily understand the consequences of those actions in the same way. Maybe they will, but this is a place where I am a little bit conservative.

AI in nuclear command, control, and communications [00:18:50]

Luisa Rodriguez: OK, I would love to talk about AI in the nuclear command and control space.

Paul Scharre: Things aren’t scary enough.

Luisa Rodriguez: Exactly, exactly. Let’s just escalate it a bit further. So nuclear command, control, and communications is this bucket that includes sensors and analysts, command nodes, communications links, and procedures used to detect nuclear attacks — and then decide how to respond, and then order and execute that response. So it’s this huge bucket that all relates to detecting and responding to potential nuclear threats. What is the argument for building AI into these systems?

Paul Scharre: I think the argument for integrating AI or automation into nuclear operations is that it’s absolutely crucial for these systems to operate at a very high degree of performance and reliability.

Within the nuclear space, there’s this concept of the “always/never dilemma.” What this means is that you always want nuclear weapons to be used if there is an authorised order from the president or whoever the command authority is for these systems; to have a breakdown there undermines nuclear deterrence. If your adversary knows that there’s holes in your system — if the president says to use nuclear weapons, maybe it doesn’t happen, or maybe they don’t launch — that might create incentives for an adversary to engage in risky behaviour.

But on the other hand, you never want nuclear weapons to be used when there’s not an authorised signal for launch, either by accident or by some rogue actor doing that.

Now, that’s really hard from an either technical or social engineering standpoint. You can imagine that there’s lots of safeguards that you can put in place against an accident or an accidental launch or unauthorised launch occurring that then make it harder for an authorised launch to occur. I think the idea is that you could use AI to sharpen this distinction if you do it right, to make things both more reliable when needed, and safer.

And on the detection side, you could use AI to have greater visibility on what adversaries are doing, and buy more time for decision makers. A fundamental problem in the nuclear space is, let’s say — and we’ve had these incidents both in the United States and in the Soviet Union, and Russia after the fall of the Soviet Union — there’ll be an alarm that goes off that looks like a missile launch. Right now, decision makers have minutes to make a decision.

The problem is that, even though both the US and Russia have invested in their nuclear arsenals in ways that ought to allow them to survive a first strike and still retaliate, you could really be severely disadvantaged by a major first strike. So there are incentives to what’s called “launch on warning.” So you have this warning that missiles are inbound: you need to launch your missiles before they get taken out in their silos, and there’s not much time.

So if you could use automation to buy more time for decision makers to speed up the process, and then make that information more reliable, that would be really valuable. So I think there is a legitimate case for using AI. And to be fair, there are lots of places where automation is already used today in actually nuclear operations. But boy, there are ways you can get it wrong too, and that’s what’s scary.

Luisa Rodriguez: Yeah. I find these arguments in favour of more automation and more AI in nuclear command, control, and communications very alluring. But I have the sense that you are keen to proceed with caution. Can you talk through one or two of the failure modes that worry you most?

Paul Scharre: Maybe a good place to start is with Stanislav Petrov, because it’s this really clear example of how things could go wrong in a really bad way.

So Stanislav Petrov is this lieutenant colonel in the Soviet military. He’s a watch officer on duty on a night in September in 1983. He’s sitting in this bunker outside Moscow, and the system says that there’s a missile launch from the United States. The United States has launched, according to the system, a nuclear-tipped intercontinental ballistic missile against the Soviet Union. And then the system says there’s another launch, and another, and another, and another: five missiles total inbound. Petrov describes that there’s this big backlit red screen, and it’s saying “MISSILE LAUNCH.”

He has really not much time to decide what I’m going to do here. Now, he knows a couple things that are outside the details of the system itself. One, he knows that the Soviet Union had just deployed a new satellite-based early-warning system, and he knows a lot of technology like this doesn’t work. So he doesn’t necessarily trust the system from the get-go. He also thinks that launching five missiles just doesn’t make sense. If the US were to launch a massive strike, you’d launch all the missiles. Why poke the bear? It just doesn’t strategically make any sense when he thinks about what the US might do.

So he calls the early-warning radars, which should be able to see the missiles coming over the horizon. They say they don’t see anything. Petrov says later that he thought it was 50/50 whether this was legit, but he had a funny feeling in his gut that it didn’t make sense. So he tells his superiors that the system’s malfunctioning.

And the scary thing about this is: what would an automated system have done? Whatever it was programmed to do. It certainly wouldn’t have known necessarily the ability to step outside that situation and say, “This is a new system: should I trust it?” Or, “This type of attack just doesn’t make any sense,” and then come to that conclusion that he did. And it certainly wouldn’t have known the stakes — which Petrov understood — that, boy, if we get this wrong, a lot of people are going to die.

I think this is a really stark, illustrative example of the stakes if we get this wrong.

Luisa Rodriguez: Yeah, I’m completely with you on the stakes, and I’ve found it incredibly unsettling to learn about all of the near misses that we’ve had. So I’m with you on that.

But — and I don’t want to be naive and fall prey to kind of magical thinking about how great AI could be in theory — it does feel to me like an automated system with quite hard-coded programming might respond to that situation in a different way to Petrov, in a way that is catastrophic. But when I chat to LLMs, it does seem like they’re able to, one, be programmed, not just programmed, but trained to be flexibly conservative. And also to have a bunch of context, including what the stakes are and how strikes are likely to play out, such that they would have some sense of the ability to reason about why one might see five warheads coming or not.

Is it naive to think that AI systems might actually be able to have enough context that they’d have made similarly good judgements to Petrov?

Paul Scharre: They might, maybe. But how reliably would be the question. A couple challenges in using AI in this.

And I think there are legitimate ways to use AI in nuclear operations. For example, using image classifiers to track mobile missile launchers. Great, great potential use case.

Luisa Rodriguez: OK, nice.

Paul Scharre: But the problem with some system making some determination is in particular what is the training data that you use for a surprise attack? We don’t actually know what that looks like. And we know that AI systems often perform quite badly when pushed outside the bounds of their training data. So if you put it with a novel situation, maybe you get something good, maybe you don’t.

There’s this really interesting example from the ’80s where the Soviets created this intelligence system called VRYAN that was designed to predict the likelihood of a surprise US attack. What it was designed to do was collect data on all of these things that the Soviets thought might be indicators that the US is preparing for a surprise attack on the Soviet Union: things like the US stockpiling blood in blood banks, the locations of senior US political and military leaders. So you could see indicators of maybe it looks like maybe they’re getting ready for something. That sounds actually like a really interesting use of automation.

What happened in practice was KGB agents were basically incentivised to generate reports and feed data into the system. The data that was coming in was just bogus, because people were judged based on going out and getting information and bringing it in, so the whole thing relied on bad data.

I think that’s an example of some of the flaws of these systems. You could imagine some AI intelligence system that’s looking at all of these different indicators — troop activity, the locations of senior political leaders, and where we see their nuclear submarines and bombers and mobile missile launchers being moved — and it sort of comes to some judgement: “OK, this is my probability.”

One of the problems is: how do we verify that that’s accurate? We can verify that a lot of other AI things are good and are performing well because we could test them in their actual operating environment. We can look at image classifiers and we can get to ground truth: What is the thing? Is it accurate? We could take self-driving cars and drive them on the operating environment. In this case, we wouldn’t have any great way to measure the baseline of just like, is it good at this at all?

And then of course, a lot of AI systems are super opaque. Let’s say your AI system says, “I think there’s a 70% probability that there’s an attack.” Why? Maybe it can tell you something, but that doesn’t necessarily mean that the story it’s telling you is accurately reflective of the underlying cognitive processes inside that neural network, of course.

So I think it seems like a really dicey way to use AI. I do think that over time, militaries are going to start to integrate in intelligence communities AI in this fashion. I think they’re likely to be very conservative though, which is probably not a bad thing in this case.

Luisa Rodriguez: Yeah, OK. I think I’m moved by the fact that you won’t be able to test these in real-world situations. Maybe at best you’ll have wargames where people are making moves that they would think they would in the real world, and you can train the system on those outcomes, but you cannot train them on real-world examples of nuclear exchanges and escalations.

Then I also just buy that… I guess in particular the Petrov situation, I can imagine a model reasoning well about that. But there are so many, many ways that all of these variables could come together, and maybe it is just pretty dicey to expect them to perform well in all of those.

It feels like it’s important that we’re comparing these systems to humans, and humans are also fallible. But is your kind of overall take like, yes, humans are fallible, but they just will be better at this for a very long time, maybe indefinitely, for reasons that maybe are a little hard to explain, but that just seems likely to be true? Or maybe they’re easy to explain. Maybe you have takes on why this will be a place for human judgement for a long time?

Paul Scharre: I do think there’s certainly value in human judgement. I think a little bit depends on how militaries or intelligence services might employ AI in different kinds of high-risk scenarios. In this case, for example, you’re trying to figure out are these indicators of a true attack or a false attack? I think a use case that I would not think is wise would be to have this AI predictive system that sort of predicts the likelihood of an attack. I just think that, for all the reasons I outlined, is not a good idea.

Now here’s a use case that might be valuable: You could have an AI system that’s fed all this information, and the question you ask it and the thing that you’ve trained it to do is make the case for me why this is a false alarm. We have a crazy number of false alarms. It’s terrifying. Somebody puts the wrong tape into the system, a lot of scary things.

And that could be an interesting use case: you’re in this moment and humans are not sure how to respond, and you turn to the AI system and the AI system is like, “Have you thought about these factors? Those are things to run down. Let’s check. Is it a training tape in the system? Is it a sounding rocket that was launched and it’s not actually an attack? Is the system malfunctioning?”

In the Petrov case, the satellite system that the Soviets deployed was detecting sunlight glinting off of clouds and registering that as the flash from a nuclear launch. So that would be an interesting use case. And you would then bias the system intentionally in a way to try to help you identify false alarms.

But I think there’s value in retaining humans for some of these types of decisions, in particular because humans understand the moral stakes. So there’s kind of this more fundamental question of what are the types of things that AI is likely to be good at? What are the types of things that humans are likely to be good at?

I think things where it makes sense to use AI would be situations where we either have or can create good data on what performance looks like, and we have clear metrics of better performance. Self-driving cars are a great example of this. It’s a hard problem, but we can collect a tonne of data, and then we can create synthetic data by running simulations to amplify that data that we get. And we can test the cars in the real-world operating environment, and there’s a clear metric of performance of “don’t get into accidents.”

There are other situations where it’s just not as clear or we don’t have good data, or the right answer might often depend on context and judgement.

To give a military example, let’s say that we have a drone looking at a person, and they’re standing in a dark alleyway and they’re holding some object in their hands. What is the thing? Well, that’s the thing where AI could be really helpful. Are they holding a rifle or holding a rake in their hands? We could build up a database of images under different lighting conditions and angles and probably get to the place where AI is better than humans.

Now, let’s say we identify the person is holding a rifle. Is that person an enemy combatant? That’s a super hard problem, because it depends on context and judgement. You know, what is enemy behaviour in this area? Maybe civilians carry rifles in this area for protection against thieves and bandits. Maybe that’s super normal. Maybe the person is carrying a shovel, but we just saw them planting an IED, so they’re an enemy combatant. That’s harder.

And you could imagine maybe reasoning systems that are plugged in, that are synthesising all this information and coming to judgements, but I would be cautious in those. For all the flaws that you have in LLMs, those are use cases that I think I would just be very cautious about.

Nuclear stability and deterrence [00:36:10]

Luisa Rodriguez: OK, setting that down for now, I’m interested in how this all affects nuclear stability and deterrence. I guess we already do have some automation, but we’ll probably be building in more in different parts of the nuclear command, control, and communications system.

Do you think there will be one effect on the game theory of deterrence, or do you think it’s just like there are going to be a million, and it depends on exactly how countries incorporate these? I guess another option is countries don’t know how their adversaries are incorporating these, and they’ll be making guesses. What does all of this do to nuclear stability?

Paul Scharre: I think there’s a couple use cases that you could see that seem likely they might increase stability. One would be just better visibility on what other countries are doing, so it makes it harder to launch some kind of surprise attack. Or if you get a signal of a launch, you have a lot of other information that you can use to verify it, that can allow you to have multiple different looks at this problem. That’s one use case that would be valuable.

Another one is if militaries just get better at using automation to make their operations and responses more reliable, and that has a stabilising effect because maybe other adversaries are less willing to contemplate doing some kind of attack. Like, “We know that they’ve shortened the response time and they’re able to get better information, and we’re not going to be able to disarm them through a first strike.”

Here’s a couple use cases that might be concerning, and I don’t really know what the net effect of this would net out to be.

One is this fear that AI enables so much transparency coupled with precision-guided weapons that it might enable a first strike to disarm opponents. I think this is super unlikely, because for this to work you have to basically get all of the weapons, or almost all of them, and then have adequate missile defences to soak up like one or two that get through. I think it’s extremely hard to pull that off in practice. It’s not enough to say that we know these are their launching sites; you need to know where are the mobile missiles, in real time, for all of those systems.

Luisa Rodriguez: Right. Submarines and trucks with nuclear weapons on the backs of them driving around countries.

Paul Scharre: Yeah, that’s a really high bar to achieve, and to get to a confidence level that somebody would feel comfortable executing for a strike.

Now, the more realistic worrying possibility is that, even though an attacker might feel like this is not plausible, a defender might feel vulnerable as a result. And then they say, “We’ve got to build more weapons, we’ve got to build more silos.” We’ve seen China engage in this massive buildup of their nuclear weapons, and I do think part of that is likely China reacting to US technological advances: drones, satellites, other systems that make China feel like they’re vulnerable and they need to increase their nuclear stockpile.

So that actually could be destabilising, either because it, in an immediate situation, creates this incentive to maybe launch because you feel like your weapons are super vulnerable; or on a longer timeline, it creates sort of an arms race instability dynamic, where countries feel like we’ve got to build more of these weapons, and then others say, we’ve got to build more too.

There’s this nasty combination right now in the nuclear space of we’re moving towards, one, a tripolar nuclear world, where the balance really isn’t just between the US and Russia; it’s the US, Russia, and China trying to build up their stockpile. US analysts are looking at this and thinking that now we need to have enough weapons to counter not just Russia, but Russia and China.

So the US increases its arsenal, then others are going to increase theirs, and you could get this more complicated arms race dynamics, coupled with emerging technologies that maybe create vulnerabilities or uncertainty — whether that’s AI, space systems, cyber systems that maybe create uncertainty among political leaders about how safe their arsenal is — and then coupled with really I think disturbingly increased political salience of nuclear weapons.

Putin in particular has done a lot of nuclear sabre-rattling in the last couple of years in the context of Ukraine, but I think the US has done a decent job of responding to in a measured way, and not overreacting to. But Putin is sort of putting nuclear weapons on the table in a diplomatic sense of sort of waving this nuclear stick around, in a way that is new and different and that might change how policymakers think about the relevance of these weapons, and maybe even make them feel more usable to some countries in some types of conflicts.

And I think that combination is a little bit scary.

Luisa Rodriguez: Yep, yep. That makes sense. If tomorrow you could choose one internationally agreed-upon binding constraint on automation and AI in the context of nuclear command, control, and communications, what would that be? What would make you feel safest?

Paul Scharre: I would love to see nuclear powers agree to maintaining human control over decisions relating to using nuclear weapons. That seems like a really low bar to clear. We all ought to be able to agree that, regardless of what we do with autonomous weapons or other things, humans should maintain control over decisions related to nuclear weapons.

And the US, it’s actually official Defense Department policy that the US put out several years ago in the Nuclear Posture Review. It was agreed to between Biden and Xi, between the US and China, during talks in the Biden administration. France and the United Kingdom have had similar public statements. So there’s some really interesting, important foundations there.

Particularly I think the US-China agreement is really significant. I’d love to see that expanded to include other nuclear powers, ideally to get a statement from the P5 — so that would be the existing countries plus Russia — and to maybe deepen that a little bit, have a little more clarity, like what does that mean internally? How do we implement this kind of guidance? What’s in bounds and what’s out? And are there things that countries could do that would be credible assurances between other nations? Because statements are one thing. What are you actually doing that might assure others that we’re actually following through on that?

I’m not totally sure what that looks like, but I think that would be a really important direction to go. And there’s already an early foundation there to build upon.

Luisa Rodriguez: My first thought hearing this is that verifiable commitments seem so important. And how do you have verifiable commitments to keeping humans in the loop in these decisions? Are there proposals for technical solutions to this, or is this just an open problem that people should think more about?

Paul Scharre: I think it’s hard to get to a place where there’s clear verifiable commitments. I don’t think it’s impossible. But I don’t have an answer here, because the answer is embedded in your software and your operations in a way that’s very different than counting missile silos. We can see the missile silos, we can count them, we know how many you have. Or submarines: it’s really hard to hide. You can hide where a submarine is underwater, but it’s hard to hide the existence of a submarine. It has to come to port.

So that seems really tricky. Countries are not going to allow other nations into their nuclear operations. There’s just way too much vulnerability there. And it’s a consistent problem in arms control: how do you manage this verifiability/vulnerability paradox of like, how do I show you what I’m doing in a way that verifies that you could see what I’m doing, but doesn’t make me vulnerable because I’m giving away too much information?

I think it’s also worth taking a step back and saying, where do we need verifiability in arms control agreements? Some arms control agreements have inspection procedures. Many of them do not, because there’s just the ability to externally observe what others are doing. Again, I think in this case that’s going to be really hard.

Another way to look at this is that it’d be nice to have some credible assurances, but you don’t necessarily need that if you’re going to do it anyways. So for the United States, for example, if this is Defense Department policy because we think it’s a good idea, then encouraging other countries to adopt that same policy is in US interests regardless of exactly what they’re doing. Because we’re going to do it anyways. It’s not like another country decided to automate nuclear launch, so it seems like a good idea. If we think it’s unwise, we’re not going to do it anyways.

So you do have some dynamic. There are some examples of other weapons like that — like biological weapons, where the US signed a biological weapons treaty and has forsworn them. And even if other countries have covert programmes — it’s probably true that some other countries might on a small scale — it’s not something still the US would want to do, because the risk of blowback is simply too high.

Luisa Rodriguez: Yeah. OK. That is slightly reassuring.

What to expect over the next few decades [00:46:21]

Luisa Rodriguez: On your kind of best guess of how things go over the next few decades, can you paint a picture of what you think wars will look like, and which parts might be really fast and then where humans will kind of stay in the loop and maybe slow things down?

Paul Scharre: Sure. So I think over the next, say five to 10 years, we’ll see militaries increasingly build more drones of different types, field them in larger numbers, and they’ll have incrementally more autonomy. We might see some isolated examples — I think it’s likely — of autonomous weapons being deployed on the battlefield, but I don’t think they’ll be integrated into military operations in large numbers.

I think we’ll certainly see more mature AI technology like image classifiers used in a wide variety of contexts by militaries to understand the battlefield. I think we’ll see a lot of what you might really just call “business process automation”: people taking things that humans are now doing in Excel spreadsheets or passing along communications and information more manually gets automated, and that speeds up the tempo of war in terms of people’s ability to compress their decision-making time.

For example, one of the things that the US Army is quite keen on is if they get some intelligence, let’s say satellite imagery of where an enemy target is, they want to be able to shorten the time it takes to put useful targeting information in the hands of artillery systems that are on the ground, so they can actually carry out a strike. Right now that’s measured in hours. You might see that compressed to minutes as that gets compressed over time. But I think you’ll still see humans still engaged in a lot of operations.

Over a longer timeline, maybe 15 to 20 years, we might see some integration of swarms at small scale for sort of tactical purposes. Maybe swarming robots being used to do reconnaissance over an area or strike targets that humans have approved. We might see autonomous weapons become more widely integrated into military operations. More AI being used to generate courses of action to support decision making.

I think we’ll probably see over that time frame more AI involved in intelligence processing, so that people aren’t just looking at an image of an object that’s been found by an AI system, for example, but actually the intelligence reports are synthesised and analysed in ways by AI before they’re given to commanders. So there are still humans making decisions, but they’re increasingly relying on information that’s mediated by artificial intelligence — which introduces a lot of weird vulnerabilities: are there biases in the system, and has the enemy found ways to manipulate it? But I think you’ll see the value in that over time.

And then maybe over a timespan of 30 or 40 years, something approaching more like some of these discussions about battlefield singularity or hyperwar, where militaries have really fully integrated AI, and we see at a much larger scale operations being done in ways that start to maybe exceed humans’ abilities to stay in the loop in terms of actually managing tactics on the ground.

I would say the exception to everything I outlined is probably what’s happening in cyberspace — where I think automation will happen much, much faster, and we’ll see cyber attackers and defenders be forced into a position where they have to cede control to machines on a much faster timescale. Because of both the ability of machines to operate in what is their native environment within machines — they’re much more capable on the internet than they are out in the real world, at least at the moment — and the tempo of operations in cyberspace being one that’s closer to things like financial trading, where the competitive pressures to pull humans out of the loop are going to happen just much sooner.

Luisa Rodriguez: Yeah. OK, I want to come back to cyberwar for sure.

Financial and human costs of future “hyperwar” scenarios [00:50:42]

Luisa Rodriguez: But before we do that, you’ve got a couple of different timelines for different scales of increases in the tempo of war. So the next five-ish years is maybe a slight increase in tempo, then you get a bigger one. And then at some point, maybe 40 or 50 years from now, maybe you do start seeing something like hyperwar.

Really concretely, how fast are we talking about? In a hyperwar scenario, are wars fought and won in the span of hours or days or weeks or months?

Paul Scharre: That’s a great question. There are still very real physical constraints on moving physical objects around that still will apply to robots and to AI.

Again, cyberspace being an exception, where you could have very fast operations. There have been botnets, for example, that have spread very quickly, cyberattacks that have taken down infrastructure. For example, cyberattacks from Russia against Ukraine have taken down Ukrainian digital infrastructure in some cases in a matter of seconds. So you could have conflict unfold in cyberspace in a matter of minutes.

But in the physical world, it takes time for missiles to fly long distances, time for aircraft to fly. AI is not going to fundamentally change those physical constraints. Maybe in the long run, people say that AI is going to make better materials, blah, blah. OK, maybe incrementally. But I think you could look at the pace of advancement of aircraft propulsion: it’s not growing exponentially; it’s very incremental improvement.

So that I think will mean things like a missile salvo would unfold over a period of hours. It might be that the scale of that missile salvo is much larger. Depending on the range, it might be that it’s 20 minutes for missiles to come in and to attack, but a back and forth of a missile exchange might take several hours. Maybe within 10 hours it’s all done and the dust settles, and one side has a dominant advantage in terms of at least that initial missile salvo.

What would be different would be maybe the number of missiles and drones that are coming in — much larger than, say, the raids that we’re seeing Russia launch against Ukraine right now, for example — to maybe thousands of drones and missiles all at once. And they might be much more cooperative, and they’re dynamic and responding in ways that force defenders to basically automate their defences.

But something like a ground invasion would still take days at minimum, for a lightning invasion, a couple weeks to unfold. So I think that there are some of these physical constraints that are very real that would still exist even in the world of AI-driven warfare.

Luisa Rodriguez: Yeah, that makes sense. Can you explain why one of the changes that you expect is bigger simultaneous attacks?

Paul Scharre: There are incentives already in an initial attack to have a massive salvo, to hit as many targets as you can at once. If you look at, for example, the US shock and awe aerial campaign against Saddam in 2003; or when the US military sort of games out a potential war fight against China over Taiwan, one of the things that the US expects was that China would launch massive missile salvos against US air bases in the region to crater runways, to blow up fuel depots, to target aircraft. And if you could make these initial attacks severe enough quick enough, you can really degrade the other side’s ability to respond. If you can crater the runways before the aircraft got off the ground, great, that’s a big one.

I think what’s different about AI autonomy is a whole bunch of factors that favour scale. Let’s talk about the way drones are used in Ukraine, for example. So let’s say you’ve got about 10,000 drones operating right now on the front lines. Well, you need 10,000 people to fly those drones. You need a lot of operators.

Now, there are advantages to having the human not in the drone: you can make the drone much smaller, you can make it cheaper, you can make it disposable. If you lose the drone, you don’t lose the pilot. Pilots can gain a lot of experience over time, so pilots who maybe are not very good at first and might have died in their first mission in a crewed aircraft, they can wreck the first 20 drones or more before they really figure it out, and it’s fine; it’s really cheap. So there are advantages there, but you’re still limited in this one-to-one ratio between pilots and drones.

Autonomy totally breaks up the dynamic. Now you could have one person launch a swarm of 100 drones, and the person just says, “Go here and perform this operation,” and the drones do that autonomously. That has major advantages in allowing militaries to just exercise command and control over larger numbers of forces, and therefore to start fielding them in ways that are more incentivised. If you have the ability to now basically control an unlimited number of drones, that changes your paradigm of like, maybe just let’s crank out a lot of these things.

There’s also other ways in which over time… So let’s talk about marginal costs for drones. Go like, really nerdy for a second. So as you field larger numbers, obviously you get better economies of scale, but there’s still some marginal cost in producing that additional drone. Drones are pretty cheap, but you’ve got to make the physical hardware.

For the software, it’s different, right? The software scales differently than hardware. It might cost a lot to develop software, but once you do, it’s basically costless to replicate software. This is why you see totally different economics in handheld devices and smartphones, for example: a phone costs a good chunk of change, several hundred dollars, even though there’s huge economies of scale, there’s like 7 billion smartphones. But the software is totally different: once I’ve got the hardware, I can download apps for free. Anybody can download an app once you have that hardware in place.

Similarly, you could see that, as more of the cognitive abilities of the drone gets from the human — humans don’t scale; we’ve got to train pilots, they’re scarce resources — begin to be embedded in the software itself, that software scales very easily, and all of those economies favour scale.

There are still real physical constraints in production — like who’s going to make all these things, in the ability to transport it to the front and get it there, in logistics of doing maintenance — but I think the dynamics will benefit scale in pretty dramatic ways.

Luisa Rodriguez: Yeah, that’s fascinating. So automated weapons are likely cheaper for a bunch of the reasons you’ve just said. It sounds like the scale goes up, so maybe the cost of an individual attack goes up a bit in that sense. But how on net does the cost of war change as we get more automated weapons? I’m interested in both financial costs but also in human casualties. Intuitively it seems like you’d get way fewer human casualties, but I’m curious if that’s right.

Paul Scharre: I do think that over time AI and automation will allow militaries to do more with less in terms of personnel. Certainly for some militaries, like the US military personnel costs are very high. That’s not true for the Russian military, for example: totally different personnel model for how they’re thinking about people. So that could allow militaries to do more with less.

The question of overall cost of militaries maybe depends a lot on what is your mental model for what the price point is set of sort of defence spending, and to what extent that’s driven by defence needs versus other factors exogenous to the Defense Department and that ecosystem — like how much Congress is willing to spend, domestic factors like political issues.

I do think these changes are likely to relatively benefit less capable actors, and that is a meaningful change in thinking about cost. So for right now, prior to drones, if you wanted air power, you would have to buy an aeroplane. Aeroplanes are traditionally very expensive, and certainly a fighter jet costs maybe $50 to $100 million, an order of magnitude. Drones are super cheap. You could buy small quadcopters for maybe a few hundred thousand bucks. It doesn’t do the same thing, but it does give you air power; it does give you the ability to recon targets from the air, find them, and even carry out very small strikes.

So that sort of lowers the price of entry into air power. And because in particular AI seems to proliferate very rapidly and software proliferates easily, I think that this relatively benefits small-scale actors who are advantaged by this.

Now, in terms of the human costs, I guess there is this idea maybe of, “We have all these drones, and then people aren’t fighting, and that’s great.” I don’t believe that’s going to be true.

The cautionary tale to me would be the invention of the Gatling gun: the inventor saw the horrific bloodshed coming from the American Civil War and thought, “What if I could have a machine that could automate firing on the frontlines? Which the Gatling gun did: it was sort of a forerunner of the machine gun, and automated the process of firing, dramatically sped up firing rates on the battlefield. His vision was that then there’d be fewer soldiers on the battlefield, save lives.

The opposite happened. It dramatically expanded the lethality of warfare. We saw huge casualties in World War I trench warfare, once machine guns had matured and were fully implemented into warfare.

So I’m not sure that automation necessarily is going to pull people back. In part, I just don’t buy this vision of futures of robot armies fighting in these bloodless battles. I think humans will still be needed to perform some cognitive tasks, and some of those are likely to be actually close on the frontlines because of challenges in long-range protected communications. It’s going to be easier to have short-range communications to control robots from relatively nearby.

But also, I just think the ugly reality is likely to be that politically, unfortunately, people will have to suffer and die for wars to end. I think that’s the practical reality. And that’s tragic and that’s kind of dark, but I think that’s probably likely to be the case.

Luisa Rodriguez: Yeah. When I imagine these robot wars fighting, I’m like, but then you’re not losing anything of value, or enough of value.

Paul Scharre: Yeah, I think that’s right. And that’s unfortunately the reality of a lot of wars. If you look at the war between Russia and Ukraine, for example, the frontlines are relatively static: they’re not moving in a dramatic way. The war has devolved into this war of attrition, where some of it’s about the economics of fielding artillery, for example, and causing casualties. But a lot of it is simply a war of suffering, of who is willing to incur more costs for longer, who wants this more.

And a lot of wars unfortunately turn out that way. It would be nice if militaries could go off and fight a battle that doesn’t involve bloodshed, or they could just game all this out on their computers that say, “Oh, clearly you’re going to win.” And when there are dramatic changes, that does happen. We see politically that when there are huge differences in power, countries generally will accede to then what stronger nations want. But not always. Sometimes small countries fight fiercely and hard for their independence. And that will to fight…

Russia’s invasion of Ukraine is a really interesting example here. Part of what happens is you can think of wars as a failure of peaceful negotiations, and it moves into a sort of negotiation through violence. One of the challenges here is that on paper, people can add up military hardware, but what’s really hard to measure is things like will to fight and morale.

On paper, Russia should have ended that war in 30 days. And a lot of military analysts, myself included, thought that’s what was going to happen. What we saw was that Ukraine is superior in all of these intangible dimensions of war: morale, will to fight, leadership, unit cohesion, corruption in the ranks. That has huge effects on the battlefield, and it’s always been the case that these human factors matter a lot. Napoleon talked about them counting three to one, I believe he said, against material factors in war. But these immaterial factors, human factors, are hard to measure.

Now the interesting question is: how does AI change that? Kenneth Payne has written some phenomenal work on this. He wrote a book I, Warbot and some other work thinking through how AI changes the psychology of war. So what does will to fight mean when you have drone swarms fighting that never get tired, right?

One of the arguments that he makes for it is an illustrative example here — and I’m not saying that I buy this, but it’s an interesting argument — that oftentimes this will to fight has benefited defenders. They’re fighting for their homeland; aggressors maybe just don’t want to be there as much. It’s very clearly in Ukraine, in terms of the balance among Russian and Ukrainian troops.

When you have AI fighting, if you take that off the table, that would seem like maybe that relatively then takes away some of the advantages of defenders, that it’s more equal and that maybe in relative terms then benefits attackers more. I don’t know if that’s valid, but AI raises some really interesting questions about changes in the psychology of war.

AI warfare and the balance of power [01:06:37]

Luisa Rodriguez: I want to go back to something you said a few minutes ago. It sounds like this might cause us to enter a world where smaller, poorer states, or even non-state actors, can actually threaten much larger militaries by leveraging these cheap automated weapons systems that are offence-dominant.

How much is that going to change balance of power dynamics? Are there going to be more wars fought because it’s cheaper to start them, including by small groups that don’t have as many resources?

Paul Scharre: Well, that’s a good question. The economics of it I feel are valid, that it relatively benefits smaller groups more.

I’ll give another example here. Ukraine basically neutralised Russia’s Black Sea fleet by sinking and damaging several warships worth hundreds of millions of dollars by spending a few tens of millions of dollars in small drone boats laden with explosives that could come in and sink a warship. I think we’re going to see those tactics copied more.

Whether that leads to more wars is tricky, because it depends a lot on what you think the mechanics are that drive wars. I think one mechanic can be if there’s a disagreement between actors about the relative balance of power. And here’s a place where I think you could argue AI on net does one or the other; I could see arguments both ways.

The argument that AI might make conflicts more likely would be that, one, it’s just a disruptive change, so there’s more uncertainty about how this is used and who’s at an advantage here. And some countries might think, “We have the AI, we can win now.” In particular, countries might feel overconfident about AI, because humans often seem to overestimate what AI can do in terms of its abilities. We see this really dramatically, unfortunately, with the early implementation of autopilot in Teslas, where there were a number of fatal accidents with people sort of over-trusting the automation. So that could be one kind of risk.

Another way that AI might make wars more likely is that, as more military capability is embedded in software and algorithms, it’s much harder to measure. You can measure aeroplanes, you can measure ships, you can measure tanks — and we say, “Look, they have three times as many aircraft as we do and twice as many tanks, so maybe we shouldn’t fight a war with them.” But when it’s algorithms, it’s really hard to know. How do we know if our swarming algorithm is better than their swarming algorithm? That’s actually really tricky. Other than like, “We’ll fight them and find out,” that’s going to be really hard. So that might lead to more uncertainty and disagreements.

One way that AI might be more stabilising is if it creates more transparency and greater ability for countries to just see what others are doing, and may make it harder to carry out surprise attacks. I think we’ve actually got really solid evidence of this. We saw this in the runup to Russia’s invasion of Ukraine, where the US, because of just greater intelligence and satellite imagery, was able to have really great visibility into what Russia was doing and then share it — in a really impressive diplomatic move, share that intelligence, declassify it, share it with European allies to get Europeans on board that this was something that was going to occur.

You could see AI just makes it harder to amass forces for surprise attack, and that takes away some of the incentives. We see a little bit of this even tactically on the frontlines in Ukraine. Despite the fact that there’s all these drones, and the drones are kind of hard to defend against, the frontlines are really static — for a lot of reasons, but drones seem to be making the frontlines more static. And one of the things that we’re hearing from people on the frontlines is there’s no way to amass forces for an assault: because they have drones overhead, they can see what you’re doing, so they know that you’re going to make an attack in this area and then they can defend against it. So it contributes to this stasis.

And so all of which is like, I don’t know, you could see arguments on either side. And a lot of it depends upon how the technology is implemented by countries.

Barriers to getting to automated war [01:11:08]

Luisa Rodriguez: You’ve described this incremental push toward more and more autonomy at some point potentially leading to very automated war, maybe something like this hyperwar image, and you think that it could take decades to get there. Why do you think it’ll take that long? Are the key barriers more like technology or more like deployment because of people wanting humans in the loop?

Paul Scharre: There’s a whole bunch of barriers. I think the biggest one is really about adoption by militaries. And there’s barriers to adoption at all of these stages.

There’s barriers at the conceptual stage, so sometimes just for militaries to conceive of a world of all of these swarms of robots and drones that are fully automated, that’s a big [adjustment]. But you have some independent thinkers that are writing about these things: John Allen and Amir Husain sort of coined this term hyperwar and have written about it. You have others. But it’s going to take time for that to be absorbed into the bloodstream of the actual decision makers inside militaries.

There are challenges in procurement and acquisition. There’s a lot of just lock-in into the system. It’s hard for new entrants to sort of fight their way through bureaucratic red tape.

And then, even when you deploy the technology, one of the things that stands out really clearly in these past examples of military technical revolutions is figuring out how do you use this? Integrating it effectively into operations is really hard.

It’s sort of like, let’s say a CEO is like, “We’re going to go all-in on AI in this company” and buys enterprise subscriptions for ChatGPT or Claude or Gemini, one of the models, for everybody in the company. “OK, here you go. Have at it.” That doesn’t actually transform your business process. What transforms your business process is people figuring out, what do I do with this thing? And experimenting and everything else.

Putting the technology in the hands of people is an important step, but it’s not the only thing. That requires a lot of experimentation, requires a willingness to change your way of fighting sometimes in ways that are deeply uncomfortable for militaries.

Militaries put a lot of emphasis on identity. You know, service members identify with their service: they’re soldiers or they’re sailors or airmen or Marines. They identify with their occupation oftentimes, so a person’s a pilot or they’re a sapper — that’s a term for an engineer in the Army — or they’re in the infantry, for example, or they’re in armour, they’re a tanker. And these identities can be really important to military.

Some of these identities are so important that they persist even after the actual occupation evaporates. For example, we call people on a Navy ship sailors. We no longer have sails. They’re not climbing the mast and working the rigs, but we still call them sailors. We have people in the US Army that we call cavalry. They don’t ride horses. They don’t even know people who rode horses. That’s so long ago. But that identity of cavalry persists. They have horse Stetsons and they have boots and the whole thing.

But in some cases, that can hinder adoption. I think one of the places we’ve seen this most clearly is the US Air Force’s struggle with drones because of this sort of salience of the pilot as an identity. So when we have pilots flying Air Force Reaper drones, they’re sitting in like a cockpit on the ground wearing a flight suit, as though they were in a cockpit of an aircraft.

The Army doesn’t care about pilots as an identity; that’s not an important thing. You walk into the Army and you’re like, “I’m a pilot,” and people are like, “All right, give me a cup of coffee.” They don’t care, right? They don’t think about it that way. They have enlisted personnel flying them. They were earlier to adopt automated takeoff and landing. They have been more open to the idea of one person controlling multiple aircraft. They call their people “operators” instead of pilots. They have more automation inside their systems, even though they’re basically using the same technology built by actually the same company, believe it or not.

What the Army does care about: the Army cares about deploying overseas. So while the Air Force has their “pilots” flying from bases in the United States for their drones, the Army, at least during the wars in Iraq and Afghanistan, would forward deploy their operators for their drones — because soldiers don’t telecommute to war. Soldiers need to be there. Boots on the ground. Even though it totally doesn’t matter.

Luisa Rodriguez: Yeah, fascinating.

Paul Scharre: So these identities. And I think that can be a hindrance to adoption, and sometimes a really big one.

Luisa Rodriguez: Yeah, it is these psychological factors rather than technological ones, or even super political ones. That’s really interesting.

Failure modes of autonomous weapons systems [01:16:28]

Luisa Rodriguez: I want to talk through a few failure modes of autonomous weapons systems. You’ve already alluded to a few. One is automation bias: the idea that humans tend to over-trust machines, especially when the machines sound confident — which I already relate to, given my experience with LLMs. It’s just really hard not to be like, “This thing seems really smart. It’s probably telling me true things.”

Another one that’s come up is this kind of brittleness: these systems can look superhuman in training, but then kind of fail catastrophically when the environment changes or when someone actively tries to fool them.

There are I think loads and loads of these failure modes. Which failure mode worries you most?

Paul Scharre: All of the above. The way that I tend to think about this is that the sort of operating parameters of these systems, whether it’s a rule-based system or one that relies on machine learning, tend to be very brittle. And when you push the system outside the bounds of its operation — either it’s in a situation where we didn’t have rules for that, or rules don’t apply, or it’s just not in the training data for the machine learning system — they tend to fail quite badly.

Humans don’t think about them that way. That’s not inherently a problem if the people employing the systems understand that limitation, and they know what the bounds of its operation are. If they know that it works in this setting and it doesn’t work in that, then fine, we can compensate for that limitation. The problem is that humans — and you alluded to this with LLMs — will often experience that a machine is good at one task and then humans logically transfer that competence to some other, very closely related task.

I think you could see this in examples of self-driving cars, where certainly early on people would see that the car was effective in driving and staying in lanes, and then sort of over-trust the automation, and assume it’s just a safe driver. If a human drove that way, we’d say they’re a safe driver. And then the car would catastrophically fail by driving into concrete barriers or parked cars or fire trucks or other obstacles, resulting in fatal incidents.

So that’s sort of my concern, and I think that there’s lots of reasons to think that we would see that happen in warfare — because the environment’s uncontrolled, we don’t know what we’re going to fight, the enemy’s going to do creative things.

If you were deploying it to a conflict like what’s unfolding between Russia and Ukraine, you could test the system and you could get good data, and maybe you have a good sense of what it’s going to be. But in situations where in peacetime you’re planning, I think we should expect that the systems will be less capable in wartime. And that’s something that militaries need to account for.

Luisa Rodriguez: Yeah. So an obvious consequence is mistakes that lead to more casualties, or mistakes that lead to strategic blunders for whoever’s weapons those are. Does this also make it more likely that there are escalations of conflicts because of mistakes that cause adversaries to think a particular thing is happening that isn’t happening?

Paul Scharre: That’s something that I’m certainly deeply concerned about. We have situations where countries will often be in militarised disputes short of war, but they have their ships and aircraft operating in close proximity. And we get incidents sometimes. We had them during the Cold War between the US and Soviet Navy and Air Force. We have had incidents between the US and China, for example.

And I think that these are dicey when there’s humans involved. I think one concern is that machines do something surprising and then we don’t even know, like, was that an intentional act on the part of the other country or not? People aren’t sure how to interpret that. That’s certainly one concern. But that concern about escalation control still exists in wartime, that a lot of times countries are still managing escalation in wartime.

We could see it very tightly in the Russia-Ukraine war. The two countries are going sort of all-out against each other, but Russia is being very careful in calibrating its escalation against NATO, and the US has been very cautious in its support for Ukraine and not over-escalating the conflict. Now we’ve had recently, for example, some Russian provocations of drones flying into NATO airspace. That seems like a very intentional act on the part of Putin: he’s trying to sort of slowly erode some of NATO’s deterrents.

But that could look very different if you had some crop of drones do something, or even strike targets, and then you don’t know whether that was intentional or not. Or even you could have drones just do it by accident and then maybe unintentionally cause escalation.

Luisa Rodriguez: I’m interested in really concretely understanding the stakes. What kinds of outcomes do you think are most worrying and people should be most paying attention to when thinking about these weapons being deployed?

Paul Scharre: I think there’s several when it comes to autonomous weapons specifically. One concern of course is that you could see much greater civilian harm on the battlefield — either because of accidents, because some autonomous weapons or a group of them strike the wrong targets and cause civilian casualties; or it could be because they strike the right targets, but they’re not correctly accounting for civilians nearby. So it’s a valid enemy tank, but it parked next to a hospital and it caused all these civilian deaths just to take out this tank. It’s not proportional in terms of the military necessity of striking that target.

It could be that there’s a slow erosion of human responsibility: that by automating attacks, humans sort of don’t feel morally responsible anymore. So even if humans are pushing the button and authorising it, humans are just less engaged with what’s occurring. That’s one concern. Certainly for countries that just aren’t concerned about civilian casualties, this could allow greater destruction.

Then in terms of escalation, my concern would be that autonomous weapons introduce another… There’s a concept of the slippery slope towards warfare. A lot of conflict short of war involves brinkmanship — where opposing leaders might be sort of dragging each other further down the slippery slope, but nobody really knows when you’re going to tip over the edge. That’s certainly what Putin is doing with these provocations against NATO, is engaging in this kind of brinkmanship.

Autonomous weapons might make this slope slipperier in ways that are hard to see and we don’t understand. Take an example like the Cuban Missile Crisis: maybe you have autonomous aircraft or drone boats or automated missile defences shoot down something, and then maybe make it harder for political leaders to walk back. Maybe even there’s an outcry domestically — you know, “Remember the Maine! We need to remember these service members who were killed, and we need to strike back and we can’t show that we’re weak.”

There aren’t really historical examples of accidental wars. There are instances where you can see miscalculation, certainly, in many cases. But I think that we could see circumstances where there’s greater miscalculation or there’s greater accidents that might escalate conflicts.

Luisa Rodriguez: Yeah, that makes sense and is deeply unsettling. Another risk that came to mind for me while reading your books is concentration of power. Could highly automated command structures make it easier for a small group — or even an AI system itself, if that AI system were both very capable and had goals that weren’t perfectly aligned with its creator’s goals — could this enable a small group or AI to seize or hold power? Because you need the buy-in and support of fewer humans at one time.

Paul Scharre: I think the directionality of that argument is right. I think the question is of the magnitude of how strong that effect would be. Dictators often are able to maintain control over larger populations with the minority of the population on their side, based on an ethnic minority or political minority. But they do need people.

Certainly one effect of this technology — whether it’s autonomous weapons in the form of robots or police robots, or AI systems to process information — maybe is that you just need fewer people. We’ve had instances where dictatorships fall because the dictator tells the troops to fire on the protesters, and they just don’t: the troops laid down their weapons and said, “We’re not going to. These are our families and community members. We’re not going to shoot them.”

And you can take away that ability for humans to just say no, right? If you added lots of autonomous weapons, you could imagine like in 1989 when the Eastern bloc had fallen, maybe there’s scenarios where it doesn’t, that allows smaller groups to hold on to power longer.

I think in the extreme case, when you think about, would it allow one individual, or a small group, or an AI system itself, to execute a coup or take over? I think it requires a little bit more of what’s the specific mechanic of that? How would that work exactly?

As we see more infrastructure come online and be more deeply integrated into cyberspace, that introduces lots of vulnerabilities and areas where groups using AI for cyberattacks — or conceivably, down the road, AI itself — could do much more harm in the real world by taking over cyber infrastructure. That could look like a power grid, that could look like water treatment plants and nuclear power plants and other kinds of things.

Certainly digital infrastructure. One of the things in the 20th century the countries would do when they had a coup is they take over the radio station. Now it’s social media platforms, other things. You could imagine AI systems doing that, and even doing it in ways that might be subtle and manipulative. Platforms are not transparent about the algorithms behind their social media platforms and what kind of content is promoted. That’s already very contested politically. In AI systems it’s worse, because the companies themselves don’t even really know what’s driving the models to generate certain types of things. It’s much harder.

So could you get weird sneaky biases? I think that’s plausible. I think to get to more really truly catastrophic scenarios, like some small group or an AI taking over a political system, you need to I think probably end up in a world that’s maybe more wired digitally than today where the trendlines are, and where military forces are much more highly automated than they are now.

And there still probably would be humans in the loop for lots of sensible reasons for militaries. I would be more worried about things like small groups of people or AI taking control of corporations which have a lot of power in modern society and are designed for this kind of thing, and that then leading to political power or influencing the information environment — or just AI systems themselves become so central to the information environment that they’re being used to manipulate information and politics.

In some way, everything in society is downstream of information flows, right? Culture, politics. If you can manipulate that, that seems like a really alarming failure mode. And then who needs to do some sort of 20th-century-style coup with the tanks? It doesn’t matter, you’re in control.

Could autonomous weapons systems actually make us safer? [01:29:36]

Luisa Rodriguez: OK, so those are some things that worry you. I’m interested in the strongest case that these weapons systems will actually make us safer and war less deadly. I guess just sticking on coups and concentration of power: could automation make these things more difficult by improving transparency and monitoring and just making it harder for things to be done in secret or in surprise?

Paul Scharre: I think there’s a broader case to be made that we are on a trajectory towards greater transparency, towards a world of increasingly radical transparency. Secrets are harder to keep than they used to be. That’s true in our personal lives, that’s true for companies, it’s true for governments and intelligence communities.

To steal secrets back in the old days, you had to get somebody into like a vault and get a hold of paper documents. Then, you know, the people that stole documents for the Pentagon Papers shoved them in their pants to sneak them out and make photocopies. Or you’ve got to get a camera in to take pictures and things.

Now everything’s digital, so if you can hack the right system, you get access to these huge troves. And we’ve seen things like WikiLeaks, these massive dumps of documents, you’ve seen hacking of things like the Chinese hack of the Office of Personnel Management — where millions of personnel records by the US government were stolen. Or the most recent Chinese telecom hack, where China’s inside the telecom infrastructure.

The digital infrastructure of modern society means that you can hack these key companies. You just have access to everything, like you’re in the pipes and everything is increasingly digital. And that’s probably only going to increase. Like this conversation, we put online, it will be able to be digested by AI. Things are more digitised, more communications in our personal lives and businesses become more digitised.

So that maybe makes it harder for anybody to do anything in secret, including for intelligence agencies and for militaries. And that could be good and bad in lots of ways, but one might argue it might make war harder because you can’t get away with stuff that people might have 50 years ago.

Luisa Rodriguez: Right, right. Yeah. So that’s one way it might make war harder, which you might think is a reason to think that it’ll also make war less common and the world more stable.

What do you think is the strongest argument that integrating AI into military systems could reduce suffering, rather than increase it?

Paul Scharre: I think there’s actually a super strong argument here. The argument in a nutshell is that humans do a terrible job at this and cause a lot of civilian deaths through war crimes, through accidents, or just through the imprecision of modern weaponry. And AI could do better. And just like self-driving cars should be able to save lots of lives on roads by just being more precise, AI could do the same to warfare: AI could enable militaries to more precisely strike military targets and not strike civilian targets.

Now, that does hinge on whether militaries are trying to do that, which is not always… To be fair, it’s not always the case. It is a war crime to intentionally kill civilians and strike civilian targets. Militaries also do that sometimes. That’s happened historically throughout war, that nations often will target civilians. And we continue to see that in modern conflicts where countries are deliberately killing civilians or certainly not trying not to.

I think the strongest case would be, if you look at the pattern of precision-guided weapons over the course of the 20th century, it has led to less civilian suffering. So in World War II, you had to basically drop massive amounts of ordinance to hit a bridge or a factory inside a city. That led to wholesale devastation of cities in Germany and Japan, because you just couldn’t actually target the infrastructures precisely as militaries might want to.

Now, in some cases you had some countries attempting widespread aerial bombing, trying to harm civilians. But the US Army and Air Force going after German cities, for example, tried to do precision bombing going after industrial targets, and it just wasn’t precise enough.

That changed with the advent of precision-guided weapons. When you get to modern day, the US military, at least with GPS-guided bombs or laser-guided bombs, can strike targets with a high degree of precision. As a result, not only are there less civilian casualties when, say, an air force using precision-guided weapons is bombing targets, but our perception of acceptable civilian casualties has changed as a result.

For example, in debates here in the United States about US drone campaigns, there became this expectation that drone strikes would have zero civilian casualties. One could argue that’s right or wrong, but it’s a huge shift from how people thought about that, say, 80 years prior. So I think that’s sort of the strong case of technology is actually on the side of greater precision, and that could lead to less suffering and war.

Is Paul overall optimistic or pessimistic about increasing automation in the military? [01:35:23]

Luisa Rodriguez: So overall, how would you describe your general feeling about this kind of radical increase in automation that you expect to happen over the next several decades?

Paul Scharre: Well, I don’t feel great about it all. I think that it really depends on how militaries use AI. And I think that there are ways for militaries to use AI that might make warfare more precise and more humane. We need to ensure that we’re adopting AI in ways that don’t lose our humanity in the process.

I think it’s important to keep humans in control of warfare, to manage escalation so that humans have the ability to end wars. And I think it’s important for humans to bear some moral responsibility for killing and suffering and war. And that’s actually a harder argument, because it’s real people that bear that cost, right? That’s someone who then is suffering from PTSD afterwards because of their effects in war.

I fought in Iraq and Afghanistan. I’ve had a lot of friends who continue to suffer after the war has ended from not just physical but mental and emotional injuries, moral injuries that they might have suffered during the war. And it’s not really fair that, as a democratic society, we make a decision as a whole to go to war, and it’s a very small slice of people that bear that cost for the war.

But I do think it’s worth asking, what would it mean for war? What would it mean for us if no one slept uneasy at night, if no one was concerned about the suffering and the casualties that occurred in war, and would that make wars more likely or more deadly? I think it’s possible to imagine a future where we adopt this technology in a way that does lead to positive outcomes, that makes warfare more precise and humane, and doesn’t lose our humanity in the process. But I do think it matters how we do it. And I think we want to be thoughtful about how we use the technology.

Paul’s takes on AGI’s transformative potential and whether natsec people buy it [01:37:42]

Luisa Rodriguez: Pushing on. We often talk on this podcast about transformative AI — so AGI, superintelligence, other terms describing highly advanced AI systems. And achieving them is the stated aim of many of the top AI firms right now.

Do you personally buy the arguments that AGI could massively transform society in the next few decades?

Paul Scharre: Yeah, I mean, I think it’s actually kind of bonkers for people not to believe that, given everything we’ve seen with the technology. We’ve been hearing for years now that deep learning is hitting a wall, it’s petering out. Maybe that’ll happen, but that prediction has been consistently wrong so far.

What we’ve seen a pattern of is an AI system will be released to great fanfare, oftentimes people sort of poke at it and realise it’s not quite as good as maybe a company had hyped it out to be, and it has a bunch of criticisms, or people will say, you know, it can’t do this, can’t do that, they can’t reason — and then six or eight months later, another AI system comes along that just totally obliterates that criticism. So given those trendlines, I think it seems quite reasonable to think that we’re going to continue to see AI improve.

And one of the things that I’m certainly struck by is these sort of “long” timelines that sometimes people have now towards AGI are often really short. So people will hear critics of AI — critics — like, “AGI’s a decade away.” That’s really close! Like, that’s crazy. I remember things from 10 years ago, it’s not that long ago.

I would say that my expectations for how fast this is unfolding have certainly been, like many people’s, pulled forward over the last couple of years. Things that, if I would have had to make a best guess, I thought maybe this will happen by 2040 are happening like now, and I’m like, wow.

I’m really struck by that, because I’ve been working on these issues for probably 15 years in various forms, in the Pentagon and here at the think tank — I work at the Center for a New American Security. I was certainly working on AI issues before people were calling them “AI” — when it was like automation or autonomy or something, before this version of the deep learning revolution really kicked off. And it strikes me that I continually am surprised by the pace of progress, and that concerns me a little bit.

I think what worries me in particular is that a lot of the technology seems to democratise violence, in particular at very extreme scales. So the questions about autonomous weapons, for example, that we’ve been talking about, or swarms and warfare are interesting. I think they’re concerning. I think there’s a lot that we ought to be doing, that nations ought to come together to put some rules of the road in place.

That’s not really the scary stuff. The scary stuff is things like biological weapons. It’s the intersection of AI and cybersecurity and biological weapons. It’s things like using AI to design much more powerful synthetic biological weapons. It’s democratising that technology as we see AI tools and large language models and more general purpose systems and agents become open source and available to anyone.

As we continue to digitise our world, more critical things become networked and vulnerable to disruption through cyberattacks: power grids, water treatment plants. Our information systems are now vulnerable to disruption digitally through hacking of telecom networks which has occurred through manipulating social media.

Those long-term trends really worry me, and I think that there are some really concerning outlier possibilities and really horrible types of catastrophic harm that could come from that. They’re probably not likely, but I kind of don’t want to find out. How likely is it that someone makes some horrible biological weapon that kills millions or hundreds of millions or billions of people? I don’t know. Let’s not inch up to that line. So those are the scenarios that really concern me.

Luisa Rodriguez: Do you think that these ideas are taken seriously in national security circles, or do they still sound kind of sci-fi?

Paul Scharre: I think generally they sound kind of sci-fi. In Washington, people have certainly got the AI bug, and that’s been true for a while that Washington is all-in on integrating AI into the US military, on maintaining US dominance in artificial intelligence overall, vis-a-vis China in particular.

You know, things like “superintelligence” or “AGI” are like a bit of a dirty word. AGI is becoming a term that’s becoming a little bit more normalised in discussion about AI, so less so. But I think it’s interesting, because the defence and security community that I come from and that I work in spends a lot of time thinking about hypothetical scenarios. We do detailed games and scenarios of nuclear war with China or Russia, or there’s a war unfolding between different kinds of countries and how does the US respond? And we do really detailed analysis, and people take those things very seriously.

So if I were to do a project, for example, and lots of experts do this, on how a major war between the United States and China might unfold over a period of maybe even years — like a very protracted conflict, and maybe there’s limited nuclear use involved — people take that super seriously. But then if you start talking about, “I’ve got this really sophisticated AI agent and it escapes from a lab and it spreads on the internet and then it hacks a critical infrastructure and takes down the power grid,” then, “What are you talking about? You’re watching too many movies.”

I’ve observed this trend long enough in the security space that it’s interesting to me that that’s the same reaction that people had to autonomous weapons or drone swarms 15 years ago — if you were talking about that, people were sort of like, “I think you watch too many Terminator movies” — and now those are taken quite seriously. So I do think that people will get there, but for some reason there is a real hangup on taking some of these scenarios seriously.

Luisa Rodriguez: Do you have a guess on what specifically the hangup is?

Paul Scharre: I don’t really know. I’ve spent a lot of time actually puzzling over this. I’ll be at conferences or in conversations with people or in roundtable discussions or private workshops, and I’ve thought this to myself a lot: what is the hangup here?

It seems like there’s something about this idea of AI reaching or eclipsing human intelligence that is really hard for people cognitively to wrap their minds around. You tend to get really weird reactions from people when you put that question on the table. I think there’s some people that just reject it as not possible, not going to happen in any reasonable time frame. It just strikes them as fanciful.

The flip side is that I’ll often see a lot of writing in this space where people get to this point of AGI and then it’s like… magic happens, right? It’s like, then it can program itself and then it recursively self-improves and then superintelligent within a period of hours or weeks or months or whatever — your vision of what that takeoff looks like — and then it can just do anything, right? Like it could take over the world and it builds these robot factories and… And you’re like, well, there are still physical constraints that exist. How exactly would those sorts of things unfold? And it’s like, “Well, it’s superintelligent. You figure all that out.” So I think sometimes you get people then reacting to that. They’re like, “That’s just nonsense.” It strikes them as fanciful.

I think there’s something about this idea, I don’t know what it is, of AI eclipsing human intelligence that seems really hard for people to just grapple with in a way that’s grounded and that takes it seriously. And I don’t know why that is.

Luisa Rodriguez: Yeah, super interesting.

Cyberwarfare [01:46:55]

Luisa Rodriguez: Pushing on: what impact will AI and autonomy have on cyberwarfare?

Paul Scharre: This is a place where I do think the effects of AI will be very dramatic and much faster than in other areas, like in physical domains. But it’s maybe worth starting where we are today, which is the effect is somewhat limited of AI and large language models, at least.

Now, automation is already widely used in cybersecurity for defensive and offensive purposes. The first self-replicating worm, the Morris worm, dates back to the 1980s. So we’ve had self-replicating malware for a while now that can spawn across computer networks.

We’ve had really quite sophisticated cyberweapons. Stuxnet — the one that is widely believed to be built by the United States and Israel, that took down Iranian centrifuges to sabotage the nuclear programme — had really sophisticated automation, so that it could spread across networks that were air-gapped from the internet, and where the people controlling them couldn’t direct what the malware was doing.

And certain forms of automated vulnerability discovery and patching have been around for several years now. I think we should expect that AI will continue to sort of advantage both attackers and defenders here by finding more vulnerabilities more effectively, by being able to recon networks to better understand what’s happening throughout the kill chain, if you will, of a cyberattack — that AI will start to play roles incrementally in each of these in enhancing human productivity.

I think the interesting question is: do we get to the point down the road where malware is much more intelligent and adaptive than today? Today you have malware that spreads on its own, that is self-replicating, that acquires resources — like botnets that acquire computing resources and then can use them for things like distributed denial-of-service attacks, can sort of leverage that. But when there are adaptations to malware, those are done manually.

Conficker, this huge worm that spread across the internet several years ago, is a really interesting case where there were a bunch of variants that evolved over time. So the taskforce that was put together — of law enforcement and intelligence communities and the private sector — to combat this worm was fighting different variants over time, but those were all designed by humans.

So do we get to the point where you have malware that’s actually able to evolve and adapt somewhat? Either it’s more clever when it’s on a computer network and able to maybe hide itself in response to threats, or adapt what it’s doing to the network itself? You can imagine a more capable reasoning model that could assess what’s going on on computer networks and then make some reasonable judgement about what to do or actually change itself over time, which would seem like a much more dangerous kind of threat.

And we’ve seen sort of concerning attempts by language models to engage in behaviour like self-exfiltration, copying itself in ways that would try to preserve its goals, or copying itself to overwrite the goals that a human would do of a new system. Now, the models aren’t very good at that yet, because it’s just not good enough yet at software engineering. But you sort of have all of the pieces in place: self-replication already exists, the ability to acquire computing resources already exists, the tendency of models — it’s not common, but it happens — where models might attempt some of these concerning behaviours like self-exploration.

It looks like right now the missing piece is just that they’re just not good enough. That’s going to get better. You can really count on this is going to get better. On what timeline, I don’t know. So I think that that’s a very troubling possibility in the long term, that you could end up with malware that maybe feels more like biological threats. During COVID we saw different variants over time, and then you’re sort of fighting against this threat that’s continuing to evolve. And that seems like a really difficult problem.

Luisa Rodriguez: Yeah, yeah. And does that have differing implications for either different types of groups or different powers, or is that kind of uniformly that cyberwarfare gets more impactful across the board?

Paul Scharre: I think it’s not obvious that it gets more impactful, because a lot of that has to do with how well defenders find ways to shore vulnerabilities inside networks. So this question of how does AI change the offence-defence balance in cybersecurity is I think a really critical one.

And I’ve seen compelling arguments from good analysts on both sides of this equation, so I think it’s worth starting with.

So today, right now, cybersecurity greatly benefits attackers. The reason why is that ultimately attackers get into computers and networks by finding vulnerabilities — by finding mistakes, bugs in code that they can then exploit.

The problem that defenders have is you have these massive codebases for an operating system on a computer or an industrial control system for some industrial plant or something. For defenders, it’s a little bit like trying to defend this castle that has these sprawling walls that stretch and snake over hills for miles, and you’ve got to cover every single possible entry point. The attackers only need to find one vulnerability, one door that’s unlocked, one secret tunnel that you didn’t know about that they can get in through. And then once they’re in, they can cause all sorts of problems, they can escalate their privileges, they can create new vulnerabilities to find other ways in.

So that’s kind of the status quo today. I think a couple of the strong cases for how this technology might benefit defenders. One is if you have AI that can be used to automatically find vulnerabilities. Finding a vulnerability and finding the patch for the vulnerability are sort of the same, right? So if you can find the vulnerability, you know how to patch it. And if defenders use this technology to run it against their networks before it’s deployed, or they’re just really much more assertive in doing that, they can automate a process that right now is manual. That can allow them to find and patch these vulnerabilities before attackers can get in, so that sort of starts to level the playing field.

Attackers can use these too, but right now what’s limiting defenders is the human cost of going through all this code, and automation really relatively advantages them. Now, that hinges a lot on do defenders actually do that? And that’s a big problem right now in cybersecurity: a lot of times it’s actually a vulnerability that we knew about, that the patch is available, but people haven’t updated their networks or their computers. That’s not always true, but that is a consistent problem. So do defenders actually do that?

The other really interesting, compelling case is that, I’m not sure that we’re there yet, but as AI gets better at writing code, we just have fewer bugs. And that actually is, I think, really compelling, because that doesn’t really hinge on defenders necessarily doing anything intentionally. It’s just that as we evolve over time to a world where maybe more and more software is just written by AI, if the AI gets pretty good at it, there just might be fewer vulnerabilities in the first place, and that actually just shores up defences. But a lot of it depends upon how the technology is employed by both sides.

Luisa Rodriguez: Yeah. That last point sounds pretty potentially huge. It does seem like things point in different directions, and it seems like it’s not clear to you that one is definitely going to dominate. But if you were to describe at least a plausible scenario that you put some stock in, where cyber capabilities do end up genuinely shifting the global balance of power or triggering escalation between major states, what is a scenario that might explain why that would end up happening?

Paul Scharre: I think that the scenario that worries me the most is that we end up in a world where malware is much more intelligent and adaptive. And you can end up with malware on the internet, like we have worms and botnets today that are intelligent, they’re able to have goals and to plan to execute them, and to adapt to defenders.

And it’s just a much more difficult problem for defenders to go against. It’s not like they’re fighting a botnet and then they could defeat it and then it’s done. It’s that this is like an intelligent and adaptive adversary in and of itself. I mean, ultimately it would have been designed and written, presumably, by humans, but whether it was used for a certain goal sometimes becomes irrelevant once the malware is launched.

It’s already true today, where there are lots of examples where an actor is trying to do something that still might not be great — like they’re trying to steal passwords, for example — and then they release some botnet that spreads across the internet because of replication and it causes all sorts of problems. Or somebody releases something open source, and now there’s all sorts of variants of this that are used by others. And that seems like a very different kind of world to be in, to be countering that kind of threat.

Luisa Rodriguez: And why wouldn’t this dynamic also be resolved or at least helped by this thing you’ve already described of defenders being able to use capable AI systems to defend against increasingly sophisticated cyberattacks?

Paul Scharre: Well, presumably defenders would be using AI to improve their defences. I think the question is: how does that net out, and why might we still have really nasty threats?

One trajectory could be that we just have a continuation of the relative balance we have today: even though people know vulnerabilities exist and it’s really important to defend against threats, there are still ways in. You know, humans are dumb and they click on the link in the email that they shouldn’t, or people don’t update their software in ways that they should. So that could be just one scenario where you just get this contest on a higher level.

One way that you could see this really sophisticated AI-driven malware benefit attackers in relative terms is if they’re just more comfortable taking risks. Then you could see maybe someone cobbles together some offensive cyber system that has an LLM as a component of it, and it’s doing some reasoning and it’s writing software, and it’s not totally clear what this thing might do, but an attacker has just a much higher risk tolerance, right? And they’re going to use it to hack some server, engage in some spam email thing or some denial-of-service attack or whatever they’re trying to do.

And defenders have the same access to the same technology, maybe even a little bit better if they have early access to models through more sophisticated labs. But they’re just more hesitant, because they’re like, “I don’t know what this thing’s going to do, and I can’t have this intelligent AI defensive system running around on my network, and then it decides that the weak link is the humans, and it locks out all the humans, or it does something strange.” And because of the unpredictability of the systems, defenders are a little more cautious in employing it, and that might benefit attackers.

My colleague Caleb Withers at the Center for a New American Security has a great report on this issue of cybersecurity and why it might benefit attackers that’s recently come out, and that’s really worth checking out.

Luisa Rodriguez: Cool. We’ll link to that for sure. Before we move on, is there anything that you feel is under-understood or underrated in the area of cyberwarfare as it gets more automated and more AI-based that you’d want to make a pitch for?

Paul Scharre: Here’s what worries me: I see two trendlines that I think intersect in some troubling ways.

One is this trendline towards more of human civilisation and our lives becoming digitised and networked and accessible through computers. That trendline seems like it’s likely to just continue: that we see exponential growth in “internet of things” devices, in network bandwidth for both wired and wireless networks.

More things are becoming digitally connected, which makes them inherently vulnerable to cyberattack. How vulnerable depends a lot on how much defenders do — but 40 years ago, you couldn’t attack a power station through the computer network. There was no way to do that. Now it’s been done. Russia has taken down elements of Ukraine’s power grid through cyberattack. So that’s been demonstrated, for example.

So we have this long trendline, and then you have this simultaneous trendline of artificial intelligence becoming more capable. And that sort of worries me that you could end up in this place where you have much more intelligent forms of malware, much more sophisticated ones, and there’s just this greater inherent vulnerability in society over time — that lots of things that we care about are actually now vulnerable, just as the systems that might hack them are becoming much more sophisticated. And that feels uncomfortable.

Luisa Rodriguez: Yep, yep. I’m unsettled.

US-China balance of power and surveillance with AI [02:02:49]

Luisa Rodriguez: Pushing onto another topic, I’m interested in how the integration of AI and autonomy into military systems shapes the balance of power between the US and China in particular. Is there kind of a high-level answer to that question before we get into some details?

Paul Scharre: Well, there are some ideas. One argument that I’ve heard is that China will be more willing to automate systems because it’s an authoritarian regime and they don’t trust their people.

I’m not sure that that’s valid. That’s a thing I hear in Washington, people saying about China. When I talk to Chinese military officers or former Chinese military officers who engage in these kinds of issues, thinking about AI in the military, I don’t really hear that. I hear a healthy scepticism about AI and a desire to adopt AI into the military, but also concerns that many humans should be on the loop. Or sometimes the Chinese translation is “above the loop”: this idea of supervisory human control. Maybe humans can’t be in the middle of everything, but we do know that these AI systems, they screw things up. We don’t necessarily trust them. We want to maintain control. There was a strong desire in the Chinese system for control from the top down.

I think one concern that I have would be about risk taking and assurances in test and evaluation of AI systems. There are pretty robust procedures in place within the US military to test new weapons systems to make sure that they’re reliable once we deploy them. The Chinese military, the People’s Liberation Army, might be more willing to take risks because they feel behind, and feel like they need to catch up to the United States.

Or if they get guidance from senior leaders to do this, they’re just going to do it. Does this drone have a high failure rate? Doesn’t matter. Xi says do it, so we need to deploy this thing. That concern about risk taking in accidents is something that I do worry about.

Luisa Rodriguez: Can you imagine AI accidents becoming the next flashpoints? Kind of the equivalent to near-miss incidents during the Cold War?

Paul Scharre: Absolutely. That’s a big concern that I have, that if you transition to a world where, say the United States and China have drones deployed at sea, in the air, undersea, interacting in contested areas like the South China Sea or near the Taiwan Strait. And they have some degree of autonomy. They don’t have to be fully autonomous weapons perhaps per se, but they have some degree of autonomy that maybe does something strange that causes an incident. So you’ve got an autonomous boat and it gets too close to another boat or causes a collision, or a drone that gets too close and causes a collision, and then there’s a mishap and there’s a political incident.

Now, I don’t really think that those incidents themselves then lead to full-scale war, not unless leaders are looking for an excuse to go to war. But I don’t know that it’s helpful to introduce a lot of this more friction and unpredictable dynamic into that situation. And there might be situations where leaders feel compelled to respond in some way to look strong, and I think that would be concerning.

Luisa Rodriguez: In the nuclear space, something roughly like nuclear parity has created a kind of uneasy stability. Is there a version of that for AI and automation parity in the military that does the same?

Paul Scharre: It depends a little bit, I think, on the scenario that we’re envisioning. One of the scenarios that US war planners really worry about is a Chinese invasion of Taiwan. And Xi, the Chinese leader, has said that Taiwan is part of China — that’s their view, and they intend to retake control by force if necessary, and has directed the Chinese military to be prepared to do so by 2027.

Now, I don’t know realistically whether they’ll be able to do that. I think probably not. But I think the key threshold in that particular scenario is if they think they could get away with it. And I think if you got to the place where they believed their technology gave them an advantage enough that they thought that they could get away with it and keep the US at bay.

And coupled with the Chinese view that the US is weak — it’s a declining power, it’s riven with domestic strife — and the US doesn’t really care that much about Taiwan: when push comes to shove, the US doesn’t have the stomach for a long, protracted war. Even if the US does get involved in the fight, China’s willing to gut it out over the long term if it comes down to a bloody fight on the island, and the US doesn’t have the stomach for that. And we could see how weak the US is in Ukraine and they’re not willing to support Ukrainians, and the US isn’t even fighting there.

So that might lead Xi to then say, “You know what? I’m gonna do it.” Similarly to Putin thinking, “I’m going to invade Ukraine. I’m gonna do it.” And in some cases from these leaders, maybe it’s a legacy issue for them, right? They see retaking territory that they see as theirs is something that they want to leave as a defining legacy. And you can have situations where then authoritarian leaders engage in some pretty risk-taking behaviour.

And even if it looks dumb to us — it’s very clear that Russia’s invasion of Ukraine has on net made Russia economically, politically, and militarily weaker, and has strengthened NATO — he still did it. And I think that would be concerning.

Luisa Rodriguez: Yeah. It seems like there are just so many parallels to Cold War-era building up of nuclear stockpiles and nuclear technology. And it seems like China has these interests that could be really benefited by having this — at least temporary, if not indefinite — strategic advantage militarily. Are the US and China currently in a race to automate and build AI into their military systems?

Paul Scharre: Well, they’re certainly in a competition militarily to maintain an advantage over the other and to adopt AI.

Sometimes it’s characterised as an arms race. It is clearly not an arms race, if you use that term in a precise academic way. The way that academics talk about arms races — and we have historical examples of the nuclear arms race, the arms race among battleship construction in the early 20th century — academics define it as above-normal levels of defence spending that’s driven by two countries competing against another.

It’s kind of hard to pin down numbers of AI spending inside militaries. Bloomberg had done some really interesting work a couple years ago poring through the Defense Department budget to try to figure this out. Like how much is the Defense Department spending on artificial intelligence? And they don’t have a good answer internally, DoD doesn’t, interestingly.

Bloomberg came up with about 1%. That’s not an arms race. That’s not even a priority. When you have senior defence leaders saying, “AI is our number one priority” — no, it’s not. Your Joint Strike Fighter is your number one priority when they look at what you’re actually doing.

So I think it’s clearly not an arms race. I do think that there is an adoption competition in AI of how do militaries find ways to import the technology. Both the US and China are going to have access to roughly the same level of AI technology. Whether OpenAI is a couple months ahead of DeepSeek just doesn’t matter, because let’s say that there’s a gap of six to 12 months between leading labs in the United States and China. Well, if the US military is charitably five years behind the frontier of AI in adoption, maybe more like 10, that one-year advantage means nothing. It’s really a contest of adoption.

But it becomes the most critical thing, figuring out how do you use this technology in a way that’s constructive, that actually advantages war fighters? And I think that’s a tricky one. It has a lot to do with how militaries organise themselves and create the right incentives internally for experimentation and reorganisation. And I think it’s just not actually clear who has an advantage there.

Luisa Rodriguez: You’ve argued that military power depends less on really excellent algorithms and more on data, compute, talent, and institutions. This is a bit of a multi-layer question, but between the US and China, where does each side actually hold an edge? I’m interested in currently and where things are going.

Paul Scharre: Right. So the question that I had at the outset of Four Battlegrounds, my most recent book, was: The US and China are in this competition. What does it mean to compete in artificial intelligence? What are the things that you’re competing over exactly? What are the sources of national advantage?

If you compare it to the Industrial Revolution, we saw that during the Industrial Revolution, nations rose and fell on the global stage based on how rapidly they industrialised. But also the key metrics of power changed: now you want to count manufacturing output, and coal and steel production, and oil became a geostrategic resource worth fighting over.

So what is that in an age of AI? Of course, at the technical level, there are three core technical inputs into AI systems: algorithms, data, and computing power, or “compute.”

When you look for a competitive advantage there, it’s not that algorithms don’t matter, but I don’t think that they are likely to lead to a competitive advantage because the algorithms are really hard to keep secret and they proliferate pretty quickly.

Now, that could change. You could end up in a world where the leading labs really go dark in terms of sharing their secret sauce, and they stop publishing papers and they stop sharing even slightly sanitised details about what they’re doing. They’re much more like pharmaceutical companies maybe, that really work hard to keep trade secrets. That might happen naturally as AI becomes more valuable, but right now we still see an awful lot of transparency from the leading AI labs, so I don’t know that that’s a source of competitive advantage today.

Data and computing power are certainly areas of competitive advantage, I think. I’ve certainly seen people make an argument that China has an advantage in data because it’s an authoritarian system. The government is putting in place all these surveillance measures to collect data on their citizens. It’s true the Chinese Communist Party is doing a lot of really dystopian surroundings, scary things to gather biometric data and genetic data on their citizens. And there’s surveillance cameras everywhere. They’re increasingly incorporating AI.

I don’t know that that nets out to a data advantage from China in particular, because what matters for companies is that the US companies are not confined to the US population. Sometimes people will say that China’s got a bigger population. Well, US companies have global reach. Facebook and YouTube have over 2 billion users each. In fact, the US companies have done a lot better job of global penetration than Chinese companies today. There’s some exceptions, like TikTok.

And the sort of collection that the Chinese government is doing doesn’t always translate into advantages for companies. Obviously some companies, like if you’re a facial recognition company, you’re going to have advantages over a US company because we don’t have the same degree of public surveillance here in the United States. Not that we should, to be clear. But the Chinese government has put in place protections on consumer data privacy, because they don’t really want Chinese companies to have the same spying powers that the government has, and they are concerned about keeping some of these big tech companies in check.

So I think it’s more of a wash, probably, on data. A lot of it has to do with how to use data better. But that’s a really interesting contested area, and I’ve heard different arguments on different sides of this.

The computing power: it’s very clear the US has a massive advantage here, partly because of Nvidia. A lot of, of course, the technical chokepoints actually exist below the stack, earlier in the supply chain at the semiconductor manufacturing equipment and software that’s used to make the most advanced chips at fabs in TSMC. Which is kind of wild, because the idea that these most advanced chips in the world are built at this island 100 miles off the coast of China, that the Chinese Communist Party has pledged to absorb by force if necessary, like, none of that’s good.

But because they rely on US technology, the US has put in place extraterritorial export controls on these chips. So right now, if a chip is designed in China by a Chinese chip design company, manufactured at TSMC, and then shipped back to China for use inside China, that’s banned by US export controls above a certain threshold in terms of chip performance.

There have been a couple incidents where maybe that’s a little bit leaky, but I think if the US can crack down on export control enforcement, that’s a clear competitive edge for the United States. Not if we just sell those chips to China, which we’ve seen some recent moves of the Trump administration to do. But I think it’s a potential advantage there for the US to harness.

People also matter a lot. Yes, if you look at the numbers, China is training more AI scientists and engineers. And in fact, China produces more of the top AI scientists and engineers than any other country, if you look at research publications in conferences, for example. But those top Chinese scientists usually don’t stay in China. Many of them come to the United States, and they come to US universities to do graduate study and stay here afterwards.

I think that’s a huge latent strength of the United States that the best scientists and engineers around the world want to come to the US to study and to work in companies. And that’s a place where the US immigration policy is just shooting ourselves in the foot, because that is very much one where, if we constrain ourselves to US-generated engineers, we will lose this competition with China. But if we’re drawing in the best scientists and engineers from the world, the US has an unparalleled advantage here, because engineers don’t want to go to China to live there.

Then I think the last key piece of this is what we were talking earlier about, about institutions and this question of adoption. I think this is more of probably a level playing field between the US and China, whether it’s in the military or other areas. How do we find ways to adopt AI that are beneficial across society?

I think it’s kind of an even question with very different political systems. Some cases that benefit China because they can move faster in many ways; they don’t have the same kind of public give and take that we have between civil society and the media and the government and the private sector. I do think that leads to better outcomes here in the United States, but that’s very much an open question.

But I do think the US has huge competitive advantages in computing hardware and talent if we harness them, and there’s more we have to do there.

Luisa Rodriguez: Do you have a prediction of which of those factors will prove most decisive?

Paul Scharre: If the current trends that we’re seeing hold, I think that the advantage that US labs have over Chinese labs is effectively negligible in terms of societal adoption. The technology proliferates way too quickly, and what’s going to matter more is how societies adopt AI to increase productivity, to increase welfare.

I know sometimes people can look at authoritarian systems and maybe be a little bit envious because they can move faster a lot of times. We saw, for example, early in COVID that the Chinese Communist Party was way more effective at closing everything nationwide and containing the spread once they got serious about it. In the US you had protests and people didn’t want to listen; it was like a much more thorny problem here.

I don’t think in the long run that authoritarian systems are better than democratic ones. Democratic ones are messier, but authoritarian systems end up being very brittle, because it ends up being whatever answer the top leadership wants. And they don’t always get it right. COVID is a great example here. Over time, China’s Zero COVID policy was counterproductive and destructive. And I do think that this give and take is a lot messier in democratic societies, but it’s probably in the long run going to lead to better outcomes.

But it does depend a lot on how we manage this transition. If AI has really disruptive effects on jobs and the labour market, are we going to take care of people? I don’t know that we’ve done a great job in the United States of managing the effects over the last several decades of job disruption from globalisation and automation, and it’s led to a lot of discontent today.

I remember in the ’90s the debates about globalisation, and at the time, what we heard from leadership was, “Sure, maybe some jobs will be disrupted and overseas, but we’ll help people reskill and get new jobs.” We didn’t really put in place those social safety nets to the extent that I think are necessary. And I think that’s going to be essential to managing this transition effectively.

Luisa Rodriguez: Yeah. How would you characterise China on that front?

Paul Scharre: Well, the Chinese Communist Party, in this current iteration under Xi, cares immensely about political stability. So we’ve seen just incredible economic growth, really unprecedented in human history, in China over the last several decades — pulling hundreds of millions of people out of poverty, increasing welfare across their society.

In previous leaders, the party had been willing to have some degree of political openness in small degrees and prioritise economic growth. We’ve seen with Xi that flips: Xi has really prioritised political control, and has been willing to sacrifice some economic growth. I think whether China is able to navigate that effectively is a really open question.

There’s this idea of the authoritarian dilemma: that basically authoritarian governments have this choice of either they can allow openness and free economic trade and open information and allow economic growth, but that will lead over time to greater political openness as well — or they can crack down and they go the way of North Korea, and you get political control, but you stifle economic growth. And actually, North and South Korea are really interesting examples of this.

I think today, China’s been able to navigate that extremely effectively. They’ve been able to basically have their cake and eat it too, and maintain political control and economic growth. And I think how China navigates that is going to be really critical to answering this question of whether China is able to continue to grow as an economic power, whether they eclipse the United States or come closer to it — or they kind of end up, as some people argue, we’ve sort of hit peak China in terms of the relative power to the United States. At least the current leadership seems to be really prioritising political control over growth.

Luisa Rodriguez: Yeah, I want to ask about this comparison in North Korea. North Korea has done a depressingly good job at its authoritarian ruling of people. China could, in theory, potentially do even more if it wanted to because of this trajectory toward really, really capable AI systems in surveillance and censorship and repression. Those could get much cheaper and more effective.

Does that seem like a plausible route China could go? I’m interested in particular about how one of our past guests, Tom Davidson, argued that AI could enable new forms of authoritarianism that can actually be really stably locked in. Does that seem plausible?

Paul Scharre: Well, I think it’s very clear that China is already building this very dystopian techno-surveillance system within the country to monitor and surveil and control its population. And the trends are in the direction of technology enabling even greater authoritarian control.

The most extreme version of this is in Xinjiang, where China has had this really intense crackdown on the Uyghur population there, and you have sort of these concentric circles of control from actual prisons that many Uyghurs are imprisoned in. For those that are released, they’re on a sort of degree of house arrest. Or even within the cities, there are physical checkpoints, police checkpoints that are doing biometric scanning, that are monitoring licence plate readers to track where people go. Knives have QR codes on them to track knives. The technology is allowing a lot of monitoring of civilians to be possible.

And then we’ve seen sort of this Xinjiangification of the rest of China, where there are surveillance cameras that are widely deployed throughout the country. When I was in Beijing last, I was really struck by this. I’d heard about all the surveillance cameras, and I didn’t really appreciate just how omnipresent they are until I was physically there. You’d be on a street corner, and they’re in every street corner, there’s multiple of them pointing in different directions halfway down the street. They’re very obvious — because, of course, the Party wants you to know that they’re watching.

So in Tiananmen Square, I roughly estimated maybe 200 cameras across Tiananmen Square. Now, Tiananmen Square is huge, to be fair. It’s massive. But I realised that the goal of the Party is not even necessarily to, if there’s another massive protest in Tiananmen Square, capture everyone there and their faces and who’s the leaders — it’s to prevent that protest from ever even happening in the first place. Protests would never even make it there, because there’s so much monitoring there.

So it’s down to things that almost seem absurd in the degree of control. China has the social credit system that is a little bit caricatured in how people talk about it. It’s not one system; it’s sort of a whole bunch of different credit scores and blacklists, some of which are for financial credit, but some of them are more social in nature. So someone might get a score based on whether they separate their trash and recycling, like down to that level of control over what people do.

And of course, technology enables really high degrees of control. So they can do things like, if you use a credit card to buy a hotel room — or maybe we don’t need a credit card, maybe we just monitor what hotels are doing — if you buy a hotel room in the city in which you live, what are you up to?

Luisa Rodriguez: Oh, god.

Paul Scharre: So they get that data and maybe they can look into this and what’s this person doing? Things like if two people are in an internet cafe at the same time every day: that’s an interesting coincidence; what’s going on there?

I think that over time that’s going to become even more. China has put in place this intense degree of control over the internet. In the ’90s people thought that was not going to be possible. Bill Clinton had famously compared controlling the internet to nailing Jell-O to the wall. Well, they did it. They did it. And it’s not that you can’t get outside of the great firewall within China through VPNs, but it’s hard and it’s kind of spotty and people just don’t do it.

So there’s so much control and so much censorship inside China, and propaganda by the Party that the Party’s been extremely effective at controlling the information environment inside China. And now they’re using technology the same in physical space, to control people’s movements and where they go and what they do. And I think technology will enable an unprecedented degree of authoritarian control.

To what extent that allows this kind of lock-in that you’re concerned about, I don’t know. But I certainly think the trendlines are very worrying in terms of enabling greater technological control.

And by the way, a lot of technology is spreading outside of China to other countries as well, and we’re starting to see this sort of global spread of more techno-authoritarianism.

It’s all bad. This is the challenge. A lot of these topics are just like, it’s scary things. And I think we’re doing a great job of trying to highlight what’s the other side? Is there a positive side? And there are, but it’s just like none of it makes you sleep easy enough.

Luisa Rodriguez: No, no.

Policy and governance that could make us safer [02:29:11]

Luisa Rodriguez: Let’s talk a bit about the policy tools and governance ideas that could make AI and autonomy safer. Some people want a global ban on autonomous weapons systems entirely. What do you think of that as a proposal?

Paul Scharre: I understand the sentiment. I think it’s just not realistic. And right now we’re on a path where there is no global regulation, there’s no rules of the road for militaries and corporate autonomy. I don’t think that’s a good position to be in. I think it’s dangerous. I think a sort of “anything” goes approach could probably take us in a dangerous place.

So we’d like to see some guardrails put in place. But I think that they have to be things that we think militaries are actually going to get on board with, and none of the major military powers have said that they support a ban.

You might be able to envision more narrow, tailored kinds of restrictions that might be achievable. One is surrounding maintaining human control over nuclear weapons, as we’ve talked about.

Another one that I think is actually interesting and worth exploring would be a ban on anti-personnel autonomous weapons. So it’s drawing a distinction between what the military call anti-materiel autonomous weapons that target physical objects — tanks, aeroplanes, radar, submarines, ships — versus targeting people directly.

I think that there might be traction there. You may have more possibility for a whole bunch of reasons. One is it’s just less militarily valuable to use autonomous weapons to go after people. I think there’s probably a case to be made that you might need autonomous weapons in some kinds of really high-intensity conflicts. If you have, say, a swarm of drones attacking enemy radars and their communications are jammed, there’s just not a good way to execute that problem to take down this integrated air missile defence system without using autonomous weapons. And keeping a human in the loop, well, people don’t move that fast, right? Outrunning bullets has not been effective since the age of the machine gun.

So I think you could make a case that going after people, you can keep humans in the loop. Now, I think we could see on the battlefield in Ukraine the case for using, at least in a military sense, anti-personnel autonomous weapons, because you have a lot of jamming on the frontlines of drones and people using drones to target individuals. But it’s not as critical necessarily for militaries.

And then I think the ick factor is a lot higher when going after people. It’s not just that it feels uncomfortable — this really does matter, in terms of getting public support — but it’s not just that “I don’t like that.” I think you can make a very reasonable argument that if you are an enemy combatant or a civilian that is in a situation where autonomous weapons are targeting you, and they’re going after a physical object, you can escape being targeted: you can climb out of that tank and run away. You can’t stop being a person. So if the autonomous weapon goes awry or you want to surrender or it’s having a mistake, I think that having weapons that target people could be really risky and cause quite a bit of harm, and it could take you on a path towards greater civilian harm.

Maybe you have countries deploying these swarms of anti-personnel drones, and they’re like, “These people are all enemy commands” — and they’re basing it on some algorithm that looks at their cell phone connections or their geolocation data or their social media. It says, “We’ve labelled them that they’re all affiliated.”

It leads to greater civilian harm. I think there’s a lot of ways in which you could say that’s much more troubling, and that might be a place for countries to explore. So I would focus more on what’s achievable here.

And I think that part of the challenge is that some of the people who have pushed for a ban on autonomous weapons are coming out of the humanitarian disarmament community, where there were a lot of successes in the ’90s at the end of the Cold War with bans on landmines and cluster munitions.

We’re in a very different geopolitical environment now, particularly since Russia’s invasion of Ukraine. Countries are now looking at landmines quite differently, certainly. And if you’re looking at saying we’re going to disarm Ukrainians and say they can’t use these weapons that might help them in the war — boy, you better have a really strong case to do that. So you’ve got to just factor in the political realities here.

Luisa Rodriguez: Yeah, that makes tonnes of sense. So that’s kind of a broad ban and then also narrow bans as policy ideas. Are there other types of proposals for major treaties aiming to govern autonomous weapons systems that you think have promise?

Paul Scharre: I think that something that involves sort of rules of the road between nations involved in air, and situations where their naval and air forces might be interacting in close proximity in crises could be very helpful here.

During the Cold War, the US and Soviet Union agreed to the Incidents at Sea agreement that helped deconflict their forces so that they were less likely to get into some of these kinds of altercations that could be dangerous and destabilising. Something like an autonomous incidents agreement, where countries might agree to put some rules of the road in place of like, how are our autonomous systems going to behave? How can we communicate effectively to you what they’re going to do?

And one might criticise that these aren’t going to hold in wartime: all the rules are going to come off. It’s not designed to solve that problem. It’s not designed to solve a wartime problem; it’s designed to solve a peacetime problem in these militarised disputes where countries don’t really want to necessarily go to war, but they’re engaging in some level of brinkmanship. So that might be something worth exploring. I think there’s value there.

I think the last thing would be maybe promulgating global norms about better safety and testing and evaluation for autonomous systems or particularly in weapons systems. Because I don’t think the accidents really benefit anyone here. So if some country develops some autonomous weapon and then it has an accident and kills a bunch of people, you know, I don’t think that’s in anyone’s interest.

So finding ways to improve the reliability of systems, and reduce access, and make countries more conscious of safety, I think is beneficial. An analogy here would be some of the work that the US does in promulgating norms on legal weapons reviews. The US and many other western militaries do legal reviews of new weapons to ensure that they comply with the law of war, with existing treaties and the Geneva Conventions. Not all countries do this, but the US and other countries have been very active in trying to spread this knowledge to others and encourage other countries to do this, and I think there’s merit in doing the same on testing and evaluation for some of these AI-enabled systems in militaries.

Luisa Rodriguez: Nice. Yeah, I like that that is a thing that a country that wanted to be safety oriented could do kind of unilaterally to reduce risk that doesn’t require rivals to cooperate. Are there other things like that, or is that a particularly good example?

Paul Scharre: I think you could argue that there are similar analogies in the risks surrounding loss of control of highly advanced AI systems. For example, just having more awareness of those risks, leading labs demonstrating the protections that they’re putting in place to be more safe to guard against agentic systems doing something squirrely and engaging in deceptive behaviour, self-exfiltrating. Those are good norms to promulgate to others. Particularly as we see not only the frontier of AI advance, but of course the technology proliferates super rapidly — so today’s frontier system is tomorrow’s commodity system.

And similarly, spreading norms around protections against biological hazards as AI becomes more capable. I think there would be areas where there’s just a lot of value in helping to ensure that other people are engaging in responsible behaviour.

Luisa Rodriguez: Nice. Are there purely technical ways to make AI use in warfare safer? So things like explainability requirements for any system used in targeting or early warning, or failsafe mechanisms that default to human control when sensors disagree?

Paul Scharre: You can imagine lots of things you could put in place.

Luisa Rodriguez: Are there any you’re particularly excited about?

Paul Scharre: A couple. So taking the analogy from stock trading and looking at things like flash crashes and trying to avoid a similar sort of scenario, I like the idea of having human circuit breakers in AI systems. So even if we end up in a world where maybe there are autonomous weapons — physical autonomous weapons or autonomous cyberweapons — that there’s some bounds on their behaviour such that if they go awry, there are limits in how bad it gets before a human has to take some positive action for things to continue.

I think there’s things that you would take to improve just reliability and reduce the risk of accidents. But I think we should assume that there will be accidents, bad things will happen — and then how do we put in place these boundaries in their behaviour or human circuit breakers so things don’t get too badly out of control? That’s something that I would like to see militaries do.

Luisa Rodriguez: Nice. Before we move on, are there any things in solutions or policy areas that you really want people to know about and consider that are maybe underrated?

Paul Scharre: In the military space specifically, or just broadly?

Luisa Rodriguez: Military, yeah.

Paul Scharre: So this is an interesting question. The US has a policy now of maintaining human control over decisions relating to nuclear use. It’s really uncertain what that means. I would love to see work inside the US military to better characterise what does that mean? And some of that work might be classified and it’s not public. That’s fine because it’s very sensitive information. But what’s in bounds, what’s out? What does an acceptable use case look like? What’s not? And over time, we might get some rules and policies and practices in place to more accurately characterise what good uses of AI and automation are in the nuclear enterprise and what aren’t.

To give an analogy here, there’s this concept today in nuclear operations of dual phenomenology for early warning systems: if we are looking at a potential missile launch against the United States, we want two completely independent ways of sensing that missile launch. For example, one could be satellites and another one could be radar systems. We have two totally independent ways of verifying it.

One could imagine something in the AI space. Let’s say I have some AI system churning through data like we were talking about earlier, and it’s coming up with some conclusion. Not only do we want some degree of auditability, explainability with the system of what it’s doing, but also maybe I have a completely independent way, a different algorithm trained on a different dataset that’s doing something similar and that I can compare those two. There might be some merit in really high-risk applications in having something like that.

Luisa Rodriguez: Cool, cool.

How Paul’s experience in the Army informed his feelings on military automation [02:41:09]

Luisa Rodriguez: OK, we have time for one last question. We haven’t talked very much about your experience in the Army. I’m wondering if there are any experiences you’ve had in that context that have informed how you think about autonomous weapons and what it means for us to be integrating AI more and more heavily into war?

Paul Scharre: So a personal example that comes to mind for me sometimes is an incident that I was in when I was an Army Ranger deployed in Afghanistan. I was part of a small recon team, and we were out operating among the mountains. There were just three of us that had gone out to patrol this sort of ridge along this line, and we saw an Afghan man approaching us along this ridgeline, coming in our general direction.

And from a distance, we couldn’t tell if he was armed or not. Maybe he had a weapon under his cloak. If he had a radio on him, we couldn’t certainly see. Maybe there were others nearby. Maybe he was scouting for somebody. Maybe he’s just a goat herder. We couldn’t tell.

We lost sight of him, and I was a little bit concerned that he might be coming up behind us. So I went and manoeuvred to ensure I could get eyes on him, and I ended up in this position where I was up above him looking down through this crack in the rocks, and he had his back to me. And I was pretty close, to be honest. I could hear him pretty clearly. He was talking, and I couldn’t tell what he was saying. I don’t speak Pashtun. I could say “stop” and that was about it. So I couldn’t tell what he was saying.

And I didn’t know if he was talking to some other people that might be nearby that were out of sight that I couldn’t see, or he was talking on a radio, for example, maybe relaying information and other fighters are going to come attack us. We’d actually seen that exact scenario happen previously, where somebody had come looking like they were herding goats as cover, but they had a radio on them and were reporting information.

So I settled into a position with my sniper rifle so that I was ready to shoot him if I saw that he had a weapon or there were other fighters and I gauged that he was an enemy combatant. I watched him for a while and I was looking for some sign. And in my head, I was sort of, you know, weighing this decision: Do I shoot this man?

Then I heard him start singing. And I just instantly relaxed, because it struck me as a bizarre thing to be doing if he was an enemy fighter reporting information about us. He probably wasn’t singing out information over the radio. And I just instantly relaxed. I thought, you know, he’s just a goat herder around here. He’s talking to himself or his goats and he’s singing, he’s enjoying the view. I watched him for a little bit longer and then ended up leaving.

And I think about that sometimes when I think about the decisions that machines might make in warfare — because I was relying on this broader contextual judgement of like, would that be a weird thing for a human to be doing in that context? And would a machine be able to pick up on that? And sometimes that sort of broader understanding, the broader context and relying on judgement, are things that AI doesn’t necessarily do very good at.

And in the big scheme of the war, that decision did not matter. It would not have changed the outcome either way in terms of the broader US campaign in Afghanistan. But it mattered a lot to him, and it mattered to me. So to me, I think about that when I think about the stakes of autonomous weapons: that people’s lives are on the line here and we’ve got to get these decisions right. And how do we find ways to use this technology that doesn’t lose our humanity, and that doesn’t cause more suffering as a result?

Luisa Rodriguez: Yeah. Thank you for sharing that. My guest today has been Paul Scharre. Thank you so much for coming on.

Paul Scharre: Thanks for having me. It’s been a great discussion.