We spend a remarkable amount of time worrying about the day artificial intelligence becomes smarter than humanity. Will Artificial General Intelligence suddenly emerge? Will machines improve themselves faster than we can keep up? Will we reach some technological singularity where human beings are no longer the most capable intelligence on the planet?
Those are interesting questions, and they may eventually become extremely important ones. But they are not the AI problem keeping me up at night today.
The much more immediate danger is far less exotic: humans are connecting imperfect AI systems to real-world capabilities, giving them authority, trusting their outputs, and sometimes deliberately using them to cause harm. We do not need AGI for AI to become dangerous. We just need software powerful enough to amplify a human mistake, a bad architectural decision, poor governance, or malicious intent.
And we already have that.
We Keep Imagining the Wrong Villain
Science fiction has trained us to picture AI risk as a machine eventually deciding that humanity is inconvenient. It is a compelling movie plot, but it can distract us from a much more ordinary engineering reality.
Most technology failures do not begin with a machine becoming self-aware. They begin when somebody builds or deploys a system without fully understanding its failure modes. A developer grants an AI agent permissions it does not need. A company automatically executes model-generated decisions without sufficient validation. An employee pastes confidential information into an AI tool without understanding where that information may go. A security team deploys an autonomous workflow without adequately constraining what it can modify.
A malicious person can also use generative AI to scale phishing, fraud, misinformation, social engineering, harassment, or malware development. None of these scenarios require consciousness, artificial general intelligence, or anything resembling a technological singularity.
They require capability, access, and insufficient safeguards.
That distinction matters because it changes how we should think about AI safety. If we convince ourselves the biggest danger is a hypothetical superintelligence sometime in the future, we risk overlooking the systems we are putting into production Tuesday afternoon.
AI Does Not Need to Be Smarter Than You to Cause Damage
Developers already understand this principle from conventional software. A shell script is not intelligent, but give it root access and a bad command can still ruin your day. A database is not intelligent, but run the wrong DELETE statement without a WHERE clause and suddenly everyone in the company knows your name.
Automation has always amplified both competence and mistakes.
AI adds an important wrinkle because traditional automation generally follows explicitly programmed logic, while modern AI systems can operate probabilistically, interpret natural-language instructions, process untrusted information, generate new plans, and increasingly invoke tools on their own. That makes AI extraordinarily useful, but it also changes the failure model.
The dangerous part is not necessarily that the model develops bad intentions. The dangerous part is that we give the model permission to do something consequential.
That is an engineering problem, and engineering problems are things we can address.
The Model Is Only One Component of the System
One mistake I often see in discussions about AI safety is treating “the AI” as if the model exists independently from everything surrounding it. It does not.
An AI application is a system. The model is certainly part of that system, but so are retrieval systems, databases, APIs, identity infrastructure, plugins, agent tools, message queues, business logic, user interfaces, monitoring systems, human operators, and downstream automation.
A model generating an incorrect answer in an isolated chat window might be annoying. The same model generating an incorrect answer that automatically executes a financial transaction is a very different risk.
The model did not necessarily become more dangerous. The architecture did.
Consider a hypothetical AI operations agent. You ask it to investigate why a production application is slow and fix the problem. That sounds convenient, but what exactly does “fix the problem” authorize?
Can the agent restart a service? Scale infrastructure? Modify a Kubernetes deployment? Change a firewall rule? Delete a database index? Rotate credentials? Drop a database? Deploy code? Send an email to customers?
The moment we connect AI to tools, the interesting safety question stops being simply, “How accurate is the model?” We also need to ask, “What can this system actually do when it is wrong?”
That is the question developers and architects should be obsessing over.
Accountability Does Not Disappear Because AI Is Involved
One of the most important lessons emerging from early AI deployments is that organizations cannot outsource responsibility to the model.
If your company deploys an AI-powered customer service system, coding agent, decision-support system, or autonomous workflow, the organization still owns the consequences of how that system behaves. The AI is part of the product and part of the architecture. It is not a separate legal or operational entity that magically absorbs responsibility when something goes wrong.
“The AI did it” is not a security model.
It is not governance either.
This becomes especially important as AI moves from generating information to taking action. The stakes change dramatically when an AI system can modify production infrastructure, send messages, execute code, change permissions, initiate payments, or interact with customer data.
At that point, familiar engineering disciplines become more important than ever. Production access matters. Permissions matter. Isolation matters. Rollback matters. Approval boundaries matter. Observability matters.
None of those lessons were invented by generative AI. We have spent decades learning them while building distributed systems, cloud infrastructure, security architecture, CI/CD pipelines, and production operations.
AI does not invalidate those lessons. It makes them more important.
The Biggest Multiplier Is Human Intent
Then there is deliberate misuse.
A malicious person does not need artificial general intelligence. They need leverage, and generative AI can provide it.
The same capabilities that help a legitimate developer write code faster can help an attacker explore vulnerabilities faster. The technology that helps a marketing team generate personalized messages can help scammers create personalized phishing campaigns. Image, audio, and video generation can empower artists and educators while also lowering the cost of impersonation and misinformation.
This is not some strange philosophical property unique to AI. Technology has always amplified intent.
The internet amplifies communication, including good communication and bad communication. Cloud computing makes enormous computing resources accessible to researchers, startups, governments, and criminals. Encryption protects journalists and banking systems while also protecting malicious actors.
Artificial intelligence belongs to the same uncomfortable category of powerful general-purpose technologies. It expands what people can do, which means some people will use it to do things we wish they would not.
The risk does not come exclusively from the intelligence of the technology. It comes from the combination of human intent, machine capability, access, and scale.
Good Intentions Can Be Dangerous Too
Malicious actors are actually the easier part of this conversation because we already know bad actors exist. The subtler danger is well-intentioned people deploying systems they do not adequately understand.
Imagine a developer builds an AI system that reviews medical claims. Nobody involved is trying to hurt anyone. But perhaps the training data contains historical bias. Perhaps the model occasionally invents policy details. Perhaps the system provides recommendations with a confidence level that users misinterpret. Perhaps over time human reviewers begin rubber-stamping its recommendations because the AI is usually right.
Nothing malicious happened, yet the system can still create harm.
This is where the phrase “human in the loop” can become dangerously comforting. Putting a person somewhere in the workflow is not automatically a safeguard. If the human does not have enough information, training, authority, time, or incentive to challenge the AI’s output, then the human may simply become an expensive confirmation button.
Safe AI is not a checkbox. It is an engineering discipline.
Treat AI Like an Untrusted Junior Operator
One mental model I find useful for AI agents is to treat them like remarkably fast junior employees who know an incredible amount, sometimes misunderstand what you asked, occasionally invent facts, and can operate your systems at machine speed.
Would you give that employee domain administrator privileges on their first day? Probably not. Would you let them transfer millions of dollars without approval? Hopefully not. Would you allow them to deploy directly to production while bypassing code review? I would certainly hope not.
Yet organizations sometimes build AI systems with essentially those capabilities because automation is exciting.
That is backwards.
The more autonomous a system becomes, the more carefully its permissions should be constrained. The principle of least privilege did not stop applying because the interface became conversational.
Build Guardrails Around Capabilities, Not Just Prompts
A lot of early AI safety work focused on prompts. We tell the model, “Never reveal sensitive data.” We instruct it, “Do not perform dangerous operations.” We add, “Always ask for confirmation.”
Those instructions are useful, but they are not security boundaries.
A system prompt is closer to application logic than an access-control list. If an AI assistant should never delete production data, do not merely tell it not to delete production data. Do not give it permission to delete production data.
If an agent needs database access for analysis, provide read-only credentials. If it needs to deploy code, require an approval workflow. If it can initiate financial transactions, implement transaction limits and independent authorization. If it generates SQL, validate the SQL before execution. If it consumes untrusted external content, assume that content may contain adversarial instructions.
If the model produces output that another system will execute, sanitize and validate that output just as you would any other untrusted input.
Developers have already learned this lesson from web security: never trust user input. AI gives us another rule to add alongside it.
Never blindly trust model output either.
AI Safety Is Mostly Good Systems Engineering
The encouraging part of this conversation is that we already know how to solve many of these problems. Not perfectly, of course, but the toolbox is familiar:
- Use least-privilege permissions.
- Separate development, staging, and production environments.
- Require human approval for high-impact operations.
- Log actions and maintain audit trails.
- Implement rate limits and defense in depth.
- Validate inputs and outputs.
- Create rollback mechanisms.
- Red-team systems before deployment.
- Test failure scenarios, not just successful ones.
- Monitor systems after release.
- Define clear ownership when something goes wrong.
- Build kill switches for autonomous workflows.
Most importantly, match the amount of autonomy you grant a system to the consequences of failure.
An AI agent that organizes your personal notes can reasonably be given more freedom than one controlling power-grid infrastructure. Risk is not determined simply by whether something contains AI. It is better understood as some combination of probability of failure, capability, access, and impact.
That sounds less exciting than debating machine consciousness.
It is also considerably more useful when you are sitting in an architecture review.
Do Not Confuse Confidence With Competence
There is another human weakness AI happens to exploit extremely well: we are easily impressed by confidence.
Large language models can produce answers that sound polished, coherent, and authoritative even when portions of the answer are incorrect. That creates an unusual interface problem because traditional software usually fails visibly. You get an exception. A button does not work. The server returns an HTTP 500.
An AI system may fail by giving you a beautifully written explanation of something that never happened.
That is harder.
As developers, we need to resist anthropomorphizing these systems. An AI saying “I verified the deployment” does not mean the deployment was verified. An agent saying “the migration succeeded” does not mean the migration succeeded. A chatbot saying “this policy allows a refund” does not mean the policy allows a refund.
Verification needs to happen outside the model whenever the consequences matter.
“Trust, but verify” is decent advice for humans.
For probabilistic software, I would shorten it to: verify.
What About Artificial General Intelligence?
None of this means research into AGI (Artificial General Intelligence) safety is pointless.
If machines eventually reach human-level general intelligence or surpass it across broad cognitive domains, that could create risks very different from the ones we are dealing with today. Those possibilities deserve serious research.
What we should avoid is pretending we know exactly when that will happen.
Predictions about AGI timelines vary enormously, and even the definition of AGI remains contested. Saying confidently that AGI will arrive next year is speculation. Saying confidently that it is decades away is speculation too.
Fortunately, we do not need to settle that argument before improving AI safety.
There may eventually be a fire-breathing dragon over the mountain.
Right now, someone left the stove on.
We should probably deal with that too.
The Responsible AI Conversation Starts With Us
Artificial intelligence is not arriving independently of humanity. We are building it. We are choosing where to deploy it. We are deciding what data it can access, which APIs it can call, and whether its recommendations remain recommendations or automatically become actions.
We are setting permission boundaries. We are deciding how aggressively to automate. Sometimes we are choosing convenience over caution because the demo looks impressive.
That means responsibility does not belong exclusively to AI researchers. It belongs to software developers, cloud architects, security engineers, engineering managers, executives, regulators, educators, and every organization deciding how these systems should operate.
The AI does not decide whether your chatbot gets access to your customer database. You do. The AI does not decide whether generated code bypasses review. You do. The AI does not decide whether a model recommendation automatically rejects somebody’s application.
Humans designed that workflow.
That should be sobering, but it should also be empowering, because unlike a hypothetical uncontrollable superintelligence, these are risks we can address right now.
Humans Are the Control Plane
Perhaps the most useful way to think about modern AI is that intelligence is becoming another programmable resource.
Cloud computing made infrastructure programmable. APIs made services programmable. Infrastructure as Code made environments programmable. AI is beginning to make reasoning, language, planning, and decision support programmable.
That is enormously powerful, but every powerful abstraction eventually runs into the same rule: someone still has to operate it responsibly.
The biggest AI safety risk today is not that a model wakes up tomorrow morning and decides to destroy humanity. It is that humans deploy increasingly capable systems without sufficient understanding, controls, accountability, or restraint.
It is the developer giving the agent too many permissions. It is the executive demanding automation before the safety architecture exists. It is the organization treating probabilistic output as ground truth. It is the employee trusting a confident answer they did not verify. It is the attacker using AI to scale something malicious.
Most of all, it is all of us becoming so fascinated by what AI can do that we forget to ask what it should be allowed to do.
Someday, we may need to solve the problem of superintelligence.
Today, we have a more familiar problem.
We need to solve the humans.
