The Six Questions Every AI Trace Should Be Able to Answer

To be useful, an AI trace should explain what the agent decided, what data it used, what it cost or whether the output was any good. Progress AI Observability Platform does this for teams building in Python, .NET. and JavaScript/TypeScript.

TL;DR

If your AI trace doesn’t explain decisions, it’s just noise. Most engineering teams ship AI agents with traces that confirm requests completed. They don’t explain what the agent decided, what data it used, what it cost or whether the output was any good.

This is the difference between trace volume and trace hygiene: having spans versus having spans that explain decisions. The Progress AI Observability Platform closes that gap for teams building in Python, .NET. and JavaScript/TypeScript. Start observing, free to start, 5-minute setup.

The Problem

After any AI incident, investigations start the same way: open the observability dashboard and look at what happened. What most teams find is a timeline of spans that confirm the request ran without errors. But they often can’t explain why the agent produced a bad answer, which piece of external data misled the model or what the interaction cost.

The traces aren’t absent. They’re answering the wrong questions.

Standard observability was designed for deterministic systems. Its core questions are infrastructure questions:

• Did the request complete?
• How long did it take?
• Was there an exception?

Those are necessary but insufficient for AI agents, where the difference between a working system and a broken one is often invisible at the protocol level. An agent can return HTTP 200 and still have reasoned over stale data, misunderstood the user’s intent, or consumed five times the expected tokens.

The questions that matter for AI agents are:

1. What did the user ask, and what context did the agent have?
2. What did external tools and APIs return—the exact data the model reasoned over?
3. What exact input did the model receive?
4. What exact response did the model produce?
5. What did this interaction cost in tokens and dollars?
6. Was the output actually good?

A trace designed to answer these is an investigative asset. One that can’t is infrastructure noise.

Trace Hygiene, Not Just Trace Volume

The AI observability community has been working through a specific version of this problem. When teams apply standard OpenTelemetry auto-instrumentation broadly, LLM spans get mixed with HTTP client requests, authentication calls, database queries and framework internals—all technically valid telemetry, none of it useful when investigating a quality failure.

Teams across the AI observability ecosystem have documented the same failure mode: a single agent trace can contain the meaningful LangChain or LLM spans alongside Microsoft Graph authentication calls, generic HTTP client activity and other unrelated infrastructure events because OTel auto-instrumentation does not distinguish AI-relevant spans from infrastructure spans.

The usual workaround is some form of instrumentation-scope blocklist or trace filtering layer, but that surface area tends to grow as more libraries get added. Across the market, vendors have responded by adding span kind selectors, annotation-based filtering, name-based filters and similar narrowing controls because isolating AI signals inside infrastructure telemetry is a recognized, ongoing problem.

The consistent pattern: trace volume is easy to produce; trace hygiene—capturing the right signals, in the right structure, with the right context—requires deliberate design.

Trace hygiene depends on five implementation principles. Those principles are what enable a trace to answer the six investigative questions above:

• Scope: Instrument LLM calls, retrieval and tool invocations—not background HTTP and infrastructure events
• Structure: Spans organized into a hierarchy that reflects agent logic, not library internals
• Content: Prompt text and completion text captured where debugging requires it
• Attribution: Tags that let you filter to a specific tenant, release or experiment on every span
• Quality: Evaluation scores connected to the traces they assess

When a trace is built on these five principles, it can answer all six questions. When it can’t, it’s noise, not signal.

Business Impact

The six questions map directly to the business capabilities that matter for AI systems in production:

Teams that answer all six questions trace AI incidents to root cause in minutes. Teams that can’t may spend hours correlating logs, reconstructing context and writing post-mortems that cite “unexpected model behavior.”

Approach

The Progress AI Observability Platform captures all six elements through automatic and manual instrumentation.

LLM calls, token counts and costs are captured automatically by the SDK. Model input and output (Questions 3 and 4) require enabling content capture—one configuration flag. External tool data (Question 2) is captured when tools are wrapped with observability. Request context (Question 1) attaches via tags. Quality scores (Question 6) come from evaluation tasks configured in the platform.

The platform supports SDKs for the most common AI development stacks:

• Python (progress-observability): One initialization call instruments OpenAI, LangChain, Anthropic, LlamaIndex and 20+ other providers automatically.
• .NET (Progress.Observability.Instrumentation): Wraps IChatClient from Microsoft.Extensions.AI and IAgent from the Microsoft Agent Framework.
• JavaScript/TypeScript: Instruments Node.js-based AI applications and frameworks.

Full documentation and setup guides for all SDKs are in the documentation.

See also: If your immediate concern is detecting AI agents that silently return wrong answers despite reporting success, see When Status OK Is Still a Failure. For managing token spend before the invoice arrives, see AI Cost Visibility Before the Invoice.

Expected Outcomes

Once traces are built on the five implementation principles—and can therefore answer all six questions—teams gain capabilities that weren’t available before:

• Root cause in minutes: The trace contains the answer—the exact prompt, the exact tool output, the exact model response. Log reconstruction is no longer required.
• Verifiable prompt changes: Validate new prompts against historical traces before deploying.
• Granular cost attribution: Spend is tied to specific workflows, tenants and model versions, not just a monthly aggregate.
• Proactive quality monitoring: Evaluation scores surface regressions before users encounter them.

Key Takeaways

• Most teams have trace volume. Fewer have trace hygiene—traces designed to answer the six investigative questions that matter after an incident.
• The gap between infrastructure observability and AI observability is a design gap made at instrumentation time, not incident time.
• Trace hygiene for AI is built on five implementation principles: Scope, Structure, Content, Attribution and Quality. Those principles are what make it possible to answer the six questions.
• The six questions every AI trace should answer: what the user asked, what tools returned, what the model received, what the model produced, what it cost and whether the output was good.
• Start observing with the Progress AI Observability Platform. A Quickstart Guide is available, it’s free to start and a 5-minute setup.

AI Observability Reading Path

1. You are here: The Six Questions Every AI Trace Should Be Able to Answer
2. Coming next: What a Useful AI Trace Should Actually Contain (And How to Build One)—for implementation walkthrough
3. NickIliev/support-agent-demo — hands-on demo on GitHub