The LLM is not doing everything. It is the last 5% of a much larger system.
The Misconception
There is a common way people talk about tools like GitHub Copilot, Claude Code, and other advanced agentic AI assistants. They talk about them as if the AI model is doing everything. As if the entire experience is just a really powerful language model thinking really hard.
That could not be further from reality.
These tools are well-engineered orchestration systems. The LLM is one component—a decision engine—called when the orchestrator needs a judgment or generated text. The rest is regular code doing regular work.
Once you see that, you see these tools very differently. And you start to understand what actually makes them powerful—and where the real engineering effort lives.
What Is Actually Happening Under the Hood
When you use an agentic AI tool to, say, refactor a module or answer a question about your codebase, here is what is actually happening at a high level:
- The orchestrator parses your request and determines what context it needs.
- Regular code searches your files, reads directory structures, filters relevant content, and assembles context.
- Regular code manages tool invocations—running terminal commands, reading files, searching codebases, calling APIs.
- The LLM is called to make a decision: which tool to invoke next, what search query to run, what code to generate, or even what additional context is needed for the next prompt of the LLM.
- Regular code takes that output, validates it, applies it, and moves to the next step.
- Repeat until the task is complete.
The LLM does not read your files. Regular code reads your files and passes the content to the LLM. The LLM does not run commands. Regular code runs commands after the LLM decides which command to run. The LLM does not manage state, handle errors, enforce safety checks, or coordinate multi-step workflows. The orchestrator does all of that.
Yes, these tools implement AI design patterns like Retrieval-Augmented Generation (RAG), Chain of Thought reasoning, tool-use loops, and context windowing. But the implementation of those patterns is regular software engineering. It is code that fetches, filters, ranks, formats, retries, validates, and coordinates. The LLM sits at specific decision points within that pipeline.
The LLM Is a Decision Engine
Think of the LLM as a decision engine. The orchestrator reaches a point where it needs a judgment — “Given this context, what should I do next?” or “Given this problem description and this code, generate the fix” — and it calls out to the model. The model returns text. The orchestrator parses that text and acts on it.
That is it.
The model is not aware of your file system. It is not maintaining a plan across steps unless the orchestrator feeds its own previous outputs back in as context. It is not choosing which tools exist or how they work. All of that is built by engineers writing deterministic, testable, well-structured code.
A single person sitting at a terminal could do the same thing manually. You could read a file, think about what to do, run a command, read the output, decide the next step, and repeat. You would be the orchestrator. Your brain would be the decision engine. The difference is speed and autonomy.
The power of agentic AI tools is not that the AI is smarter than you. It is that the orchestrator automates the loop—reading, deciding, acting, observing—at a speed and consistency no human can match.
Why This Matters
Understanding this architecture changes how you think about these tools in several important ways.
It explains their limitations
When an agentic AI tool makes a mistake, it is not always because the model is stupid. It can often be that the prompt or orchestrator fed incomplete context, chose the wrong tool, or parsed the model’s output incorrectly. The engineering around the model matters as much as—or more than—the LLM model itself.
It explains why different tools feel different
GitHub Copilot, Claude Code, Cursor, Windsurf—these tools often use the same underlying models but produce very different experiences. That is because the orchestration layer, the context management, the tool definitions, the prompt engineering, and the workflow design are all different.
The model is a commodity. The orchestration is the power.
It demystifies what you are building with
You are not working with magic. You are working with a well-designed software system that happens to call an LLM at decision points. That means you can reason about it, predict its behavior, understand its failure modes, and learn to work with it more effectively—just like any other tool.
It shows where the real engineering effort lives
If you are building AI-powered tools or integrating AI into your own applications, this is the key insight: 95% or more of the work is regular code. It is the file readers, the parsers, the search indexers, the tool definitions, the safety validators, the context assemblers, the retry logic, the state machines, and the output formatters. The AI model is the last 5% that brings intelligent decision-making to an already well-built system to really bring out its usefullness.
The Orchestrator Is the Power
I think this framing helps cut through a lot of the noise around AI tools today.
People get caught up debating which model is best — GPT-5.5 vs. Claude Opus 4.7 vs. Gemini — as if the model is the only thing that matters. But in the context of agentic tools, the model is interchangeable. What makes one tool better than another is the quality of the orchestration: how well it gathers context, how intelligently it sequences actions, how gracefully it handles failures, and how effectively it presents results.
The model provides the intelligence. The orchestrator provides the usefulness.
Without the orchestrator, you just have a chatbot. With the orchestrator, you have an autonomous agent that can navigate a codebase, run tests, fix errors, and ship changes—all triggered by a single prompt.
What This Means for Developers
If you are a developer watching the AI space, this should be encouraging. The skills that make agentic AI tools work—software architecture, systems design, API integration, state management, error handling, and thoughtful UX—are exactly the skills you already have or are building.
AI is not replacing the need for good engineering. It is creating an entirely new category of engineering where the challenge is: How do you build systems that leverage an LLM effectively?
That is an orchestration problem. And orchestration has always been a software engineering problem.
The AI brings the magic. But the code makes it useful.
