Prompt Engineering Is Not Enough

Reliable AI takes more than good prompts. Retrieval, tools, memory and evaluation help keep outputs grounded and consistent.

A team ships its first AI feature: a support assistant behind a chat panel, powered by a prompt the team spent two weeks refining. It performs well in testing.

A month after launch, the complaints arrive. The assistant recommends a pricing plan that was retired last quarter, invents a configuration flag that has never existed, asks a returning customer for details they provided the day before and tells another user it has no way to update their ticket. The team revises the prompt, and the same failures return in new wording.

This cycle is common, and it comes from asking the prompt to do a job it cannot (fully) do. In the previous article of this AI Engineering Basics series, we looked at how AI-powered applications are architected, with the model sitting as one component behind our own backend.

This article picks up the other half of that story: why production systems are built from several components working together, and why better instructions, valuable as they are, cannot substitute for the rest.

What Prompt Engineering Does Well

Prompt engineering is the craft of writing the instructions a model receives with each request: defining the assistant’s role, setting boundaries on what it should answer, fixing the output format and supplying examples of good responses.

Getting those instructions right makes a measurable difference in quality, and no amount of surrounding infrastructure makes up for instructions that are vague or contradictory.

A well-structured set of instructions for a support assistant might look like this. The example uses the OpenAI Responses API with gpt-5.6, and the same ideas apply with any provider:

const response = await openai.responses.create({
  model: "gpt-5.6",
  instructions: `You are a support assistant for a project management product.
  Only answer questions about the product and its documented features.
  If you are not confident in an answer, say so and offer to connect the user with support.
  Keep answers under 150 words. Format multi-step instructions as numbered lists.`,
  input: userMessage,
});

Each line does a specific job. The first sets the role, the second and third set boundaries, and the last fixes length and format. Instructions like these produce more consistent output than a bare “answer the user’s question,” and the skill of writing them stays relevant through everything that follows. The limits show up in what instructions cannot reach.

Where Prompts Hit Their Limits

Four gaps sit beyond the reach of any wording.

The model’s built-in knowledge is fixed. A model knows only what was in its training data (the text it learned from before release), and that data ends at a cutoff date. Our product’s current pricing and policies are not in it. The assistant recommended the retired plan because it answered from the only information it had.

The context window caps each request. We could paste documentation into the instructions, and small teams often start there. The approach stops working as content grows, both because the window fills up and because every request pays for those tokens whether they are relevant to the question or not.

The model cannot act. It produces text. Checking an order or updating a ticket requires an operation against a real system, and no instruction grants that ability.

Nothing persists between requests. The model is stateless, a property we covered in the modern AI stack article. Unless our application resends the history, yesterday’s conversation may as well have never happened.

From Prompt Engineering to Context Engineering

The shift in production systems is from perfecting a static prompt to engineering what reaches the model on every request.

Context engineering is that discipline. For each request, the backend assembles the instructions, the relevant passages from our content, the results of any operations and the facts worth remembering. It then sends the package to the model. The prompt becomes a structure our code builds at runtime rather than a string someone polishes.

Each gap has a component that closes it:

One final component rounds out the set. Once a request is assembled from this many moving parts, we need a way to confirm the whole thing works, which is where evaluation comes in.

The Components in Brief

Each gets a dedicated article later in the series, so a short introduction suffices here.

Retrieval

Our content gets indexed ahead of time, and on each request the backend searches that index for the passages relevant to the question and places only those in the prompt.

This is the Retrieval-Augmented Generation (RAG) pattern we walked through in the previous article, and it addresses the first two gaps at once: answers come from current material, and the context window carries only what the question needs. Platforms like Progress Agentic RAG provide this layer as a service.

Tools

A tool is an operation our backend exposes to the model with a name and typed parameters, plus a description of when it applies. The model requests a call, and our code validates and executes it. The result then goes back into the context for the next step.

We covered tool definitions and the runtime loop in the previous article, and the Model Context Protocol (MCP) standardizes how these connections are described.

Memory

Short-term memory is the running conversation, resent with each request so the model can follow references to earlier messages. Long-term memory is a set of facts our application chooses to save in ordinary storage and retrieve into context later, like a customer’s plan tier or their preference for weekly summaries.

The model remembers nothing on its own; our application decides what persists and when it comes back.

Evaluation

The same input can produce different outputs from one run to the next, so “it looked good when I tried it” proves nothing. An evaluation set pairs representative inputs with criteria for judging the outputs, and it runs whenever the instructions, the model, the retrieval setup or a tool definition changes. It plays the role a regression suite plays in conventional software, and it is the component teams most often discover too late.

Workflow Design: Many Small Steps Over One Big Prompt

The other habit that fades in production is the mega-prompt: one giant set of instructions asking the model to classify the request, find the right policy, reason through the answer and format the result in a single pass. Reliability drops as instructions pile up, and when something goes wrong, there is no way to tell which part failed.

Production systems decompose the work instead. In a support triage flow, one small model call classifies the incoming message and plain code routes it: billing questions pull the billing policies while bug reports fetch the matching runbook. A second call drafts the reply, and anything above a risk threshold goes to a human first. Each step can be tested on its own and swapped for a cheaper model where quality allows.

Wrap-up

Prompt engineering is where almost every team starts, and it’s a skill worth learning well. The problem is that the hardest production failures have nothing to do with the wording.

Teams that ship reliable AI features spend most of their effort on what surrounds the prompt: retrieval to keep answers grounded, tools to make actions possible, memory to carry context forward and evaluation to prove the system works after every change.

The good news for full-stack developers is that assembling components is our home turf. Later articles in this series give each of these components the depth it deserves, from memory patterns to building evaluations for non-deterministic systems.

For more on building AI-powered applications with Progress, check out the following resources:

• Progress AI Engineering Solutions
• Progress Software: AI-Powered Applications
• Progress Agentic RAG