The new UK Joint Doctrine Publication (JDP) on Unmanned Aircraft Systems has been covered in The Verge and other media under headlines like “UK government says humans will always be in charge of its robot weapon systems.”
So here we go, what does “in charge of” mean? Actually this new document repeatedly insists on the precision of its definitions and its own authenticity as a reflection of actual UK policy. So what is it saying so precisely?
The JDP doubles down on the position of its predecessor document, the 2011 Joint Doctrine Note (JDN) now marked on every page with loud announcements of its expired status, that “autonomous” must be distinguished from “automatic” or “automated” and that an “autonomous system” means one “capable of understanding higher level intent and direction.”
Defining “fully autonomous weapon” as something “with the ability to understand,” and with capabilities that stem from this “understanding,” places autonomous weapons firmly in the realm of artificial intelligence of a kind that so far exists only in science fiction. This makes it easy for the JDP to declare that
“The UK does not possess fully autonomous weapon systems and has no intention of developing them.”
However, while defining autonomy in terms of “understanding,” the JDP does not clarify what “understanding” means, other than that it provides a capability:
“From this understanding and its perception of its environment, such a system is able to take appropriate action to bring about a desired state. It is capable of deciding a course of action, from a number of alternatives, without depending on human oversight and control, although these may still be present.”
We may question what level of artificial intelligence is required in order to begin to provide capabilities that might be described in such language. In fact, existing and emerging missile systems, for example, decide a course of action from a number of alternatives without depending on human oversight and control, which are assumed not to be present. But there is clearly much further to go in AI before equaling or exceeding human capabilities for situational awareness, comprehension and direction of appropriate action in relation to a desired state.
The predecessor document, the 2011 Joint Doctrine Note (JDN), noted that
“…proportionality and distinction would be particularly problematic, as both of these areas are likely to contain elements of ambiguity requiring sophisticated judgement. Such problems are particularly difficult for a machine to solve and would likely require some form of artificial intelligence to be successful. Estimates of when artificial intelligence will be achieved (as opposed to complex and clever automated systems) vary, but the consensus seems to lie between more than 5 years and less than 15 years, with some outliers far later than this.”
Artificial intelligence seems here to have been reified as something human-like at least. If it were as little as “5 years” away (which would be a year ago), one might think it would be something to be concerned about today. However the JDN stated that “Autonomous systems will, in effect, be self-aware” and
“their response to inputs indistinguishable from, or even superior to, that of a manned aircraft. As such, they must be capable of achieving the same level of situational understanding as a human.
The point of setting such a high bar for true autonomy was that it excluded everything anyone is actually doing or planning to do:
“This level of technology is not yet achievable and so, by the definition of autonomy in this JDN, none of the currently fielded or in-development unmanned aircraft platforms can be correctly described as autonomous.”
The new JDP drops the words “self-aware” and “same level of situational understanding as a human” but retains the argument that the “technology is not yet achievable,” so nothing the MoD is doing involves autonomous weapons, case closed:
“Fully autonomous weapons systems as we describe them (machines with the ability to understand higher-level intent, being capable of deciding a course of action without depending on human oversight and control) currently do not exist and are unlikely in the near future.”
Rather, the JDP asserts, on the premise that automation is not autonomy, that
“While some companies and research organisations are trying to develop autonomous systems, the UK’s view is that increasing automation, not autonomy, is required to improve capability.”
Yet it also describes how “automation” evolves into “autonomy”:
“For example, a mission may require a remotely piloted aircraft to carry out surveillance or monitoring of a given area, looking for a particular target type, before reporting contacts to a supervisor when found. A human-authorised subsequent attack would be no different to that by a manned aircraft and would be fully compliant with the LOAC, provided the human believed that, based on the information available, the attack met LOAC requirements and extant rules of engagement.”
In text that was cut from the new version, the JDN continued:
“From this position, it would be only a small technical step to enable an unmanned aircraft to fire a weapon based solely on its own sensors, or shared information, and without recourse to higher, human authority. Provided it could be shown that the controlling system appropriately assessed the LOAC principles (military necessity;humanity; distinction and proportionality) and that ROE were satisfied, this would be entirely legal.”
Here the JDN sought to defend the legality of what would obviously be a fully autonomous hunt-identify-kill mission, with machines even doing LOAC assessments. The new JDP drops this defiant stance and limits the scenario to human-authorized but possibly machine-directed attack, calling this “greater assistance to pilots and operators, and in-system survivability in non-permissive, contested and congested battlespace.”
The JDP concludes that
“the UK does not possess armed autonomous aircraft systems and it has no intention to develop them. The UK Government’s policy is clear that the operation of UK weapons will always be under human control as an absolute guarantee of human oversight, authority and accountability. Whilst weapon systems may operate in automatic modes there is always a person involved in setting appropriate parameters.”
However, it has defined “autonomous” as something that is beyond present technology, that nobody is doing. So its disavowal of autonomous weapons places no restrictions and has no effect on anything the UK is doing or plans to do in “autonomous weapons” as the rest of the world understands the term.
We may be encouraged to know that “there is always a person involved in setting appropriate parameters” but this does not set a high standard for human control.
Nevertheless, the JDP contains some statements which may be encouraging if taken out of context, such as that
“the UK opposes the development of armed autonomous systems.”
“The UK does not own, and has no intention of developing, autonomous weapon systems as it wants commanders and politicians to act as the decision makers and to retain responsibility.”
It acknowledges the problem of predictability, and that the acceptance of responsibility by human operators and commanders for using an autonomous weapon system
“has an implicit assumption that a system will continue to behave in a predictable manner after commands are issued; clearly this becomes problematical as systems become more complex and operate for extended periods…. In reality, predictability is likely to be inversely proportional to mission and environmental complexity.”
From this candid observation, the JDP reaches a remarkable conclusion:
“For long-endurance missions engaged in complex scenarios, the authorised entity that holds legal responsibility will be required to exercise some level of supervision throughout. If so, this implies that any fielded system employing weapons will have to maintain a two-way data link between the aircraft and its controlling authority.”
If, as the document repeatedly insists, this represents actual UK policy, the conclusion that maintenance of “a two-way data link” is required for “long-endurance missions engaged in complex scenarios” may come to be of importance, as it would be one concrete standard against which the legality of systems and their use could be judged. However, a footnote weakens the pledge by adding that
“this link may not need to be continuous.”