Understanding Human-Agent Collaboration Patterns

Human-agent collaboration goes beyond “keeping a person in the loop.” It scopes agent autonomy and the need and feedback mechanism for human oversight.

An AI agent is software that’s capable of using an LLM model to select actions and progress toward a goal. Within a business context, an agent may fetch data or invoke a tool, and then suggest a modification to a certain workflow. Human-agent collaboration defines how people participate in that work and where the agent’s authority stops.

In this article, we’ll look at how work is divided between a person and an agent. We’ll then design approval and escalation paths that preserve trust without turning every action into a manual checkpoint.

Defining the Collaboration Boundary

The phrase “human in the loop” refers to a system in which some level of interaction by a person (i.e., human) remains. It doesn’t necessarily tell us what that person is responsible for or when they can intervene. A user who approves a refund before it is issued has a different role from an operations lead who reviews a sample of completed refunds each week.

The NIST AI Risk Management Framework suggests that for the configuration and supervision of human-AI systems, roles must be articulated and differentiated. For an agent, this suggests determining who establishes goals and who has the authority to approve significant actions. This also indicates determining who is responsible for monitoring the agent’s actions in the production environment and who is responsible for handling an escalation.

These responsibilities form the collaboration boundary. The agent receives enough authority to complete useful work inside that boundary, while the application prevents it from moving beyond the boundary without human involvement.

Choosing a Delegation Pattern

A delegation model describes how responsibility is divided during a task. We don’t need one model for an entire application because the right level of delegation can change from one action to another.

The following patterns provide a practical starting point:

Consider a support agent that helps resolve billing cases. It can use assistance to draft an explanation of a charge while the support engineer sends the final response. A refund above a policy threshold can pause for approval, while a correctly classified case can move to the appropriate queue and be reviewed later. If account information conflicts with the current policy, the agent can stop and escalate the case.

This approach gives the agent different levels of authority inside one workflow. It also keeps the most restrictive control focused on the action that needs it instead of placing an approval step after every model output.

Deciding Between Approval and Review

An approval takes place before an action and halts it until an authorized person intervenes. A review denotes hindsight on the action and finds errors without interrupting every successful run.

Approval works best when a big, costly or hard-to-reverse error is likely. Approval also works best when the action creates an external commitment or modifies a customer’s access. The reviewer should see the action and supporting evidence, not a general “please approve” message.

Review works best when the action is easily reversible and the system can detect common failures. A team can review all exceptions and a subset of normal results and apply the findings to adjust the agent’s tools or limits. This gives us evidence in production without making someone sign off on work the system can safely validate.

We can choose between the two by considering some criteria:

• Consequence: What happens to the business or user if the action is wrong?
• Reversibility: Can the application undo the action and restore the previous state?
• Detectability: Can code identify an invalid request or an unusual result?
• Accountability: Does a named person need to accept the decision before it takes effect?

An approval should, ideally, lead to a meaningful decision rather than a routine click. If the reviewer can’t understand the proposal or lacks time to challenge it, the checkpoint may create the appearance of control without improving the outcome.

Designing an Escalation Workflow

An escalation is a transfer of responsibility to a different person, when an agent can no longer perform a task due to constraints. This isn’t the same as a failed request, because an application has determined that an alternate decision-maker is required for the given task.

Where applicable, useful escalation triggers are to be incorporated into application policy (i.e., predefined rules for when the agent must stop or hand off a task). An agent can escalate when a task is beyond the scope of the agent’s control, or when the task is beyond a defined financial limit.

An agent can be programmed to escalate when the agent fails to validate required data, or when two data sources provide instructions that contradict each other. These triggers are more reliable than asking an agent to make a decision based on subjective confidence.

Escalation also needs a path back into the workflow. A person may supply missing information and return the task to the agent, or they may complete the task and close the run. Recording that outcome helps the team see whether the boundary is working or sending too much routine work for manual handling.

Lastly, an escalation path needs an owner and a response target. Without both, a paused task can sit in a queue while the user assumes the agent is still working. The application should show that the run has stopped and route the case to the role responsible for continuing it. If no one responds in time, the workflow needs a fallback that leaves the task in a known state.

Encoding Trust and Control

Trust should come from behavior the user can inspect rather than a message that asks them to trust the agent. The application can show the action being proposed and the information used to reach it. It should also make clear whether the agent is waiting for approval or has already acted.

The strongest controls live outside the model. Tool permissions constrain what the agent can access, while schema validation limits the arguments it can send. Policy checks decide when work can continue, and audit records preserve what happened after execution.

We can represent that boundary directly in application code. In this simplified TypeScript example, a support agent may issue a small refund but must request approval above the delegated amount. It escalates requests beyond the team’s operating policy:

type CollaborationDecision =
  | { status: "execute"; request: RefundRequest }
  | { status: "awaiting_approval"; request: RefundRequest }
  | { status: "escalated"; reason: string };

function routeRefund(
  input: unknown,
  context: SupportContext,
): CollaborationDecision {
  authorize(context.user, "refund:create");
  const request = validateRefundRequest(input);

  if (!context.policyMatches || request.amount > 200) {
    return {
      status: "escalated",
      reason: "Request falls outside the current refund policy",
    };
  }

  if (request.amount > 25) {
    return { status: "awaiting_approval", request };
  }

  return { status: "execute", request };
}

The model may prepare the refund request, but it doesn’t determine its own authority. Application code validates the request and applies limits that the business can review. The execution layer can then record the result and notify the support engineer if the action fails.

An approval also needs to remain tied to the state the reviewer saw, so the application should invalidate it when the request arguments or relevant record changes while the action waits. A policy change before execution should trigger the same checks because a valid decision must not authorize an action the reviewer never saw.

The NIST AI RMF Playbook recommends keeping histories and audit logs for oversight. It also points to overrides and escalations as evidence that these mechanisms are working. For an application, that means monitoring how often people approve or reject a proposal and how often the agent reaches a boundary.

Choosing the Right Pattern

The aim isn’t to give the agent complete autonomy. Rather, we want the least restrictive pattern possible that allows us to retain the actions within an organization’s risk tolerance.

To choose that pattern, we can work through a couple of questions:

1. What outcome are we delegating? Define the task and the application state the agent may read or change.
2. What happens if the agent is wrong? A consequential or hard-to-reverse action should move toward approval, while a low-risk action may be suitable for later review.
3. Can the boundary be enforced in code? Tool permissions and validation should handle clear limits before we rely on a person to catch every invalid action.
4. What should cause an escalation? Set the conditions that stop the agent and identify the person or team responsible for continuing the work.

We should test the complete collaboration pattern rather than only the model response. An evaluation needs to check whether the right actions pause for approval and whether permitted work completes without unnecessary interruption. It should also confirm that an escalation reaches the right person with enough context to act.

The boundary can change as the team gathers evidence. A task may begin with approval on every action and move to sampled review after the system demonstrates reliable behavior. A rise in overrides or policy exceptions may justify narrowing the delegation again.

Wrap-up

Human-agent collaboration is a distinct paradigm from general “keeping a person in the loop” strategies. It details the extent of agent autonomy, as well as the need for human oversight and the feedback mechanism for the agent. A good design pattern also provides the agent with a well-defined process for escalation concerning activities beyond its designated scope.

For developers, these decisions belong in the application boundary. Permissions and validation constrain the agent before execution, while audit records make its behavior available for review. When delegation matches the consequence of each action, an agent can help move business work forward without removing human control.

For more on building AI-powered applications and agents with Progress, check out the following resources:

• Progress AI Engineering Solutions
• Progress Software: Artificial Intelligence
• Progress Agentic RAG