What I Wrote About Rogue AI in 2016, and What Changed

12 Nov, 2026

Long before I thought about artificial intelligence as a design problem, I was losing to it in a 1999 sci-fi classic called Quake III Arena. The game offered five difficulty levels that ran from I Can Win to Nightmare, with Bring It On, Hurt Me Plenty and Hardcore in between. Bring It On was my obvious choice.¹ On the higher levels survival came down to seconds instead of minutes. The reason was hand-built game AI. Rules, state machines and pathfinding, tuned carefully by designers and programmers, made the bots sharper at gauging a player's behaviour. By Nightmare, the final fight against Xaero could be over in moments.

There was a whole roster of bot opponents, and they were frustratingly good. They seemed to read the map as I did, anticipate where I would appear next, and adjust their tactics when I changed mine. A strategy that worked twice would fail the third time. I was more irritated than impressed, and only later did I understand that the irritation was the interesting part. Something I had not built and could not fully predict was responding to me.

That experience stayed with me through art school and through years of designing dashboards and data interfaces for large organizations. It stayed with me into the graduate application I wrote in 2016.² By then the bots were a fond memory, but the lesson had become a question. If a game from the last century could make me feel read, what would systems built for far more serious purposes do once they could act on their own?

Machines respond to the world we build around them. The unsettling part is how little attention we pay to who builds that world and why.

What I Wrote in 2016

In 2016 I applied to a graduate program in strategic foresight with a proposal built around HASI, short for Human-focused Artificial Super Intelligence. Its argument was simple. Artificial intelligence was about to gain influence faster than most people were planning for, and the sensible time to decide what it should serve was before it began to act independently.

A few definitions will help. By artificial intelligence I mean systems that perform specific tasks once reserved for people, such as recognizing images or recommending products. Artificial general intelligence, or AGI, would handle a wide range of tasks as flexibly as a person does. Artificial super intelligence, or ASI, would surpass people at most of them.³ My 2016 text moved between these terms without saying so, and that is the first thing I would fix. It also said the bots in that game learned, when in fact their designers had tuned them by hand.

What the statement got right was the worry. It noted that companies were building systems "aware" of human tasks and that some of this work served military ends, and it pointed to researchers who warned against an arms race in autonomous weapons. It asked which human skills would be transformed, established or compromised, and what that would mean for cultures around the world. It also argued that people should design guardrails in advance, using design thinking and foresight, so that the goals of machines would stay aligned with our own.

While I was writing the statement I shared a draft with an experienced practitioner whose work I respected. The review was generous and direct. It warned that I had chosen an enormous subject, since a superintelligence would stand so far above us that we could grasp its abilities no better than an ant grasps ours. It noted that only two paragraphs of my statement described the research idea while most of the rest described me. It also proposed the name "hASI," for human-focused ASI, which became HASI. I have kept all three contributions. The first shaped the limits I describe later in this essay, the second is why most of this essay is about the idea and not the credentials, and the third gave the idea its name.

What it lacked was a mechanism. I asserted that design thinking and foresight could keep machines aligned with human goals, yet I never explained who would do what, at which stage, or how any of it differed from what engineers were already attempting. "Aligned goals" described a destination without a route. The rest of this essay is my attempt to draw one.

What the Decade Got Right, and What It Changed

The part of my statement that holds up best is the worry about weapons. A year before I wrote it, AI and robotics researchers signed an open letter, joined by figures such as Stephen Hawking and Elon Musk.⁴ It warned that a military AI arms race would become nearly unavoidable once any major power began developing AI weapons. Its remedy was a ban on offensive autonomous weapons "beyond meaningful human control," which is a statement about where human judgment must remain. A decade later that is still the right question.

What I could not have foreseen was how the debate would move. In 2016 I pictured a rogue superintelligence, a machine that broke free from the restraint. What arrived instead, from late 2022, were language models that millions of people now use every day, and their risks were quieter. The same review had urged me to consider narrow AI as a rich field of design challenges that could be addressed immediately. A decade on, much of the practical design work has indeed been done with systems that fall short of general intelligence. Alignment research became a central part of the field, and governments began writing rules. The European Union's AI Act came into force in August 2024 and has been phasing in its obligations since.⁵ None of that resembles the scenario I described. All of it supports the premise beneath it, which is that guardrails are something people must decide deliberately.

The nearer danger is less a machine that escapes us than people who gradually stop exercising the skills needed to oversee it.

The quieter risk is the one my own text touched in a single line, when I asked which human skills would be transformed, established or compromised. A 2025 survey of 319 knowledge workers by researchers at Microsoft and Carnegie Mellon University found that 62 per cent reported thinking less critically when they used generative AI, particularly for routine tasks.⁶ The authors warned that handing off routine work denies people the practice that maintains their judgment. The study is self-reported and cross-sectional, so it shows a pattern worth examining and not a proven decline. Even so, it suggests that the nearer danger is less a machine that escapes us than people who gradually stop exercising the skills needed to oversee it.

Humans in the Front Seat

HASI, as I wrote it in 2016, rested on three foundations. Design thinking supplied the discipline of understanding people before building anything for them. Strategic foresight supplied the habit of examining futures before committing to one. Innovation supplied the means of turning the result into something real. What I never did was say how the three would work together on an actual system. This is the structure I should have given it, and it is where my consulting framework, RACE, becomes useful.

RACE moves through four phases named Research, Advisory, Conceptualize and Educate. In each of them one HASI foundation leads.

Research is design thinking at its most patient. Before anyone sets limits on a system, someone has to learn who it will affect, which human skills it will rely on or replace, and what "human-focused" means for that deployment. It is the empathize and define work I first learned on graphic design briefs, applied to a system that will act with some independence.

Advisory is where foresight comes in. Together with the people affected and the engineers who own the system, we work through scenarios of how it could fail or drift from its purpose. Then we decide how much autonomy it receives. What may it decide alone? What must a person approve? What is it never permitted to do? These are design decisions with owners, and they are made before deployment instead of being discovered afterward.

Conceptualize is innovation in the practical sense. The agreed limits become things people can use, such as override points, escalation paths and interfaces that separate signal from noise so that an overseer sees what matters in time to act. These are prototyped and tested with the people who will do the overseeing.

Educate completes the cycle and is, I think, the most neglected phase. A guardrail that nobody is capable of using is decoration. This phase keeps the relevant human skills practised, explains to everyone affected what was decided and why, and feeds what has been learned into the next round of Research.

The difference from technical alignment work is one of target. Alignment research shapes how a model behaves. HASI shapes the relationship between the system and the people around it, which means who can intervene, with what information and with how much skill. The two are partners and not rivals. I still use the name HASI with its super-intelligence horizon because that is where the consequences would be greatest, but nothing in the method waits for that day. It applies to the systems people are deploying now.

Roles, Not Mastery

If a superintelligence cannot be designed for, the next best option is to decide what it is for. HASI therefore adds a second constraint to the first. Beyond keeping people capable of oversight, it limits the roles an advanced system is allowed to occupy, and the roles that suit it best extend human capability without replacing human judgment. Drafting an email, generating an image, cleaning and enriching a data set, or summarizing search results are tasks of that kind. A person remains the author of the result and answers for it.

In my earlier piece on the human-AI relationship I described four ways of deploying AI, which I called augment, assist, automate and autonomy.⁷ They are not stages of maturity. They are choices made in context. HASI turns those choices into a policy by giving every role a ceiling. Some roles can safely reach automation, and a few, such as search or navigation, may approach autonomy once the consequences of an error are bounded and a person can still audit the outcome. Others should stop at augmentation or assistance for good.

The harder question is which roles belong in that last group, and this is where the Advisory phase of RACE does its work. With the people affected, HASI would map the roles where a capable system could endanger livelihoods, security or public order, such as running critical infrastructure or making consequential decisions about people without review. It would recommend that those roles stay closed to the system, and that existing models be kept from acquiring the capability to fill them. I want to be careful about that second half. Design can identify the roles and make the case, but keeping a model from acquiring a capability is a task for engineers and regulators. The policy also assumes that the roles are ours to assign, which is a large assumption for a system that surpasses us. I return to that limit below.

The Skills Question

Of everything in my 2016 statement, one line has become more relevant each year. I asked which human skills would be transformed, established or compromised. I did not have a good example then, but I think I had one all along.

In my consulting work I designed a dashboard for pharmacists in a healthcare application, built so that they could understand performance analytics faster. The brief was about speed of comprehension, and the discipline behind it was separating signal from noise so that the figures needing a pharmacist's attention stood out. What made that design responsible, I now think, was less the data it showed than the judgment it left to a person. The tool shortened the path to understanding without removing the decision from the person accountable for it.

An override means little if the person who holds it has lost the knowledge to know when to use it.

Set that beside the survey of knowledge workers discussed earlier. The pattern it describes is what happens when the second half of that design principle is dropped. A system handles the routine and the person is left with the exceptions. The practice that would have prepared them for those exceptions quietly disappears, and when an unusual case finally arrives the person overseeing the system is its least prepared member.

This is why I treat skill retention as a guardrail in its own right and not as a training afterthought. An override means little if the person who holds it has lost the knowledge to know when to use it. In HASI terms, the Research phase asks which skills a system will absorb. The Advisory phase decides which of them must deliberately stay with people. The Educate phase keeps those skills practised even when the machine could do the work faster. A little friction is the price of a person who can still say no.

Where HASI Falls Short

A model that claims to put people in the front seat should say plainly what it cannot do. HASI is a way of designing the relationship between people and the systems they use, and that has limits.

The strongest objection came from that early review, and it still stands. A species that cannot grasp what a superintelligence is capable of cannot confidently design for it, and anything we asked of such a system would use only a fraction of its ability. The review offered several ways to meet the problem, and two stand out. One was to treat a constrained, human-focused branch of superintelligence as a given, which sets aside the question of how likely it is and is where the name HASI comes from. The other was to move to a narrower kind of intelligence where design can work today. I now think the second route is the honest one. The method in this essay applies to the systems being deployed now, and for superintelligence it offers a hope and not a plan.

It cannot replace technical safeguards. Containment, testing and security engineering belong to the people who build the systems, and no workshop substitutes for them. It also cannot prevent people from choosing to switch guardrails off. That is a question for governance and regulation and not for design. Participation has its own costs. It takes time and can be dominated by whoever is loudest or best funded, and it raises difficult questions about how much of a system's weaknesses can be shared with outsiders. Foresight does not predict. It prepares us for a range of futures and will miss some that matter.

A species that cannot grasp what a superintelligence is capable of cannot confidently design for it.

Finally, HASI is still a proposal. It has been an idea of mine since 2016, but I have not tested it on a system at the frontier, and I would rather say so than imply a track record I do not have. What I can say is that RACE has held up on organizational problems of real complexity and that bringing affected people in early is a logic that travels well. Whether it holds at the scale of advanced AI is a question for the next decade, and I would welcome the people who can test it.

What to Do Now

For practitioners and organizations the practical steps are modest, and none of them needs a new technology.

First, map who a system affects before deciding how much freedom it receives, including the people whose skills it will absorb. Second, write the limits of autonomy down as decisions with named owners that cover what the system may do alone, what needs a person's approval and what it must never do. Third, rehearse failure before launch with the people who will oversee the system taking part, so that awkward scenarios are discussed while they are still inexpensive to fix. Fourth, protect the skills that make oversight real by keeping practice, manual modes and clear explanations in place even when the machine could handle the work faster.

A worked example, the pharmacy

Consider a community pharmacy that introduces an AI system to verify prescriptions by cross-checking them against patient records, drug interactions and dosing ranges. Pharmacists are often told that such tools will subsume their profession. The evidence so far points to a quieter risk. Trade coverage describes the pharmacist's role moving from managing product toward clinical judgment, and it warns of automation bias, the habit of following a machine's suggestion over one's own judgment.⁸ That is the situation the four steps are meant for.

First, map who is affected. The pharmacists carry verification skill and professional accountability, the technicians enter and fill prescriptions, the patients depend on both, and the prescribers receive the system's queries. The skills the system absorbs are the routine checks, such as interaction screening, through which a pharmacist builds judgment.

Second, write down the limits and their owners. The system may count, pre-fill, flag and rank without asking. It needs a pharmacist's approval before any prescription is verified or released. It must never release a high-risk medication, or override a flagged interaction, on its own. The pharmacist in charge owns the verification rule, the operations lead owns exceptions, and the vendor owns reporting on how the model performs. Naming owners matters because regulators in Canada and the United States favour principles over fixed rules, and responsibility for an AI-assisted error remains unsettled.⁹

Third, rehearse failure before launch. Pharmacists, technicians and a patient representative walk through scenarios such as the system missing an interaction, flagging so much that staff stop reading the flags, or going offline at the busiest hour of the day. Each scenario ends with a named person and a manual fallback.

Fourth, protect the skills. Some verifications are done by hand, a sample of checks is completed before the pharmacist sees the system's suggestion, and every flag explains why it fired, so that the pharmacist is still the one exercising judgment.

In terms of the four levels, clinical verification stays at assist, while counting and inventory can move to automate. The result is a pharmacy that is changed on purpose and not subsumed by default.

Designers have a particular part to play. We are rarely present when decisions about autonomy are made, yet every day we shape how people see, trust and override these systems. That is a role worth asking for.

I began with a game whose bots could defeat me in seconds. They were not clever by themselves. People decided how hard they would hit, and the menu let me choose how much I could take. The question for the next decade is who controls that setting for far more consequential systems, and whether they still understand what it does.

Let's talk about a complex problem.

If you're navigating a transformation, enterprise design challenge, or organisational problem where the answer isn't obvious, let's explore it together.

Get in touch →
Connect on LinkedIn →

Footnotes

#ai #design-thinking #human-ai #inclusive-design #research #systems-thinking