Add Observability to a Genkit AI Agent with Progress Agent Engineering

Observability gives us a window into how an AI agent is working throughout its run, allowing us to improve its performance. Here’s how to add observability to a Genkit agent with Progress Agent Engineering.

Building an AI agent is easier today than it was a few years ago. We can connect a model, give it a prompt, add a few tools and get a useful answer quickly.

The difficult part starts after the first successful demo. What happened inside the agent? Why did it choose that answer? Which tool was slow? Did the prompt cause the problem? Is the model using too many tokens? Where is the request spending money?

As a frontend developer, I am used to opening DevTools and adding a console.log. I can check a network request and quickly see a 200 or a 500 response. That gives me a good starting point when a web application has a problem.

An AI agent is different. The request can be successful, but the answer can still be wrong, slow or too expensive. A 200 response only tells us that the request finished. It does not tell us what the model did, which tools it called or why it made that decision. At that point, our usual DevTools are not enough.

Adding a model is not only a model problem. There are several pieces to check:

• The prompt may not give the model enough context.
• A tool may return incomplete or incorrect data.
• The model may call the wrong tool or call it more than once.
• A second model call may change or misunderstand the tool results.
• The full request may use more tokens and cost more than expected.

To understand these problems, we need to see more than the final answer and more than a network status code. We need visibility into the complete agent run. We need observability.

How We Will Explore Observability in a Run

The best way to learn is with a working example. Today, we’re going to build a Genkit travel agent. Genkit helps us build the agent, connect it to Gemini and give it tools.

Our example is a Family Travel Planner. It receives a destination, the number of adults and children, the children’s ages and a budget preference. It then creates a structured travel plan with flights, a hotel, activities and an estimated cost.

We will run the same agent before and after adding observability. This will give us a simple baseline and help us see what observability adds to the application. For our run, the agent will plan a family trip to Ibiza and use three tools for flights, hotels and activities. We’ll keep the logic unchanged to compare two runs fairly.

By the end, we will have a native Genkit trace in an observability dashboard. The trace will show the generation, model calls, tool calls, token use and cost.

The Scenario

Imagine that you join a company that has started adding AI to its product. The team has a travel agent that creates plans for families. It uses a model, a prompt, and tools for flights, hotels and activities.

At first, everything works well. Then users start reporting that the agent is giving worse answers. Someone says that the model is not working anymore. Another developer remembers that the prompt changed yesterday. The data used by one of the tools was also updated this week.

Now we have several possible causes:

• The model may be having a problem.
• Someone may have changed the prompt.
• A tool may be returning different or incomplete data.
• The agent may be using more model calls than before.

How can we know where the problem is? Looking only at the final answer is not enough. We need to see the complete request and compare each step.

Why Do We Need Observability?

Console logs can tell us that a request started or failed. They usually do not show the complete relationship between the model and its tools. Observability gives us that context in one trace.

With observability, we can:

• Find the exact model or tool span that is slow.
• Inspect tool inputs and outputs when the final answer is wrong.
• See how many input and output tokens a request uses.
• Track the cost of individual requests and the application over time.
• Find failed steps without reproducing the complete request locally.
• Compare traces after changing a prompt, model or tool.

What problem does this solve for us as developers? It turns “the agent gave a bad answer” into a more useful question, such as “Did the hotel tool return incomplete data?” or “Did the second model call receive the tool results?”

When Should We Add It?

Observability is especially useful when an AI feature has more than one moving part. Consider adding it when:

• An agent calls tools or other services.
• A request includes multiple model calls or steps.
• Response time, token usage or cost matters.
• You need to investigate errors in development or production.
• You are changing prompts or models and want to compare results.

For a very small experiment that makes one model call, a console log may be enough. For a real application, observability should be added early. It gives us a baseline before traffic grows and before debugging becomes harder.

The travel planner is a good example because one request combines three tools and one or more model calls. That makes the difference between a final answer and an inspectable execution path easy to see.

Why Not Use Only the Model Provider’s Tools?

Genkit and model providers such as OpenAI can provide useful logs and monitoring features. These tools are a good place to start, especially when we are testing one provider and one simple model call.

Things get harder when the application grows. An agent can use a framework, several tools and multiple model providers. If each provider gives us a different view, we have to jump between dashboards to understand a single request. We may see the model call in one place and the tool call somewhere else.

This can also tie our observability to one provider. If we change the model, add a new provider or move from Genkit to another framework, we may need to change our monitoring setup too.

That is why an observability platform that is independent from the model provider and the application framework can help. The goal is to keep one view of the complete request, even when the technology behind the request changes.

What Is Progress Progress Agent Engineering?

Progress Agent Engineering is an observability platform that helps us see what happens inside an application while it runs. For an AI application, it collects information about model calls, tool calls, errors, duration, token usage and cost. It then presents that information in a dashboard where we can inspect a request from beginning to end.

Think of it as a flight recorder for an AI agent. The final travel plan is the result, but the trace shows the complete journey: which tools were called, which model calls were made, how long each step took and what data each step produced.

Progress Agent Engineering can be used with applications built in TypeScript/JavaScript, Python and .NET. The idea is the same in each case: add the SDK to the application, collect the important events and inspect them in the dashboard.

That is the approach we will use here. The agent runs with Genkit and Gemini, but Progress Agent Engineering gives us one place to inspect the full execution. We use the Progress TypeScript SDK because this example is a Node.js application. The same idea can be applied with the Python or .NET SDKs in applications written in those languages.

A Few Observability Terms

Before we look at the features, let’s make the vocabulary simple.

• Think of a trace as the complete receipt for one agent request.
• A span is one line on that receipt, such as a model call or a tool call.
• Instrumentation is the code that watches those operations and creates spans.
• Telemetry is the information produced by that instrumentation and sent to Progress.
• Content tracing means that prompts, responses and tool data can also be recorded. It is useful for debugging, but it can contain private information, so use it carefully.

What problem does observability solve for us as developers? Instead of guessing why a request was slow, expensive or incorrect, we can inspect each span that produced the result.

Progress Agent Engineering Features Used in This Project

Progress Agent Engineering includes several observability features. In this tutorial, we use these features:

• Tracing: Groups everything that happens during one request into a single trace. This includes the Genkit flow, model calls and travel tools.
• Native Genkit span processing: Understands the spans created by Genkit and connects them to the trace. We do not need to add a new span for every operation.
• Trace details: Lets us open a trace and inspect its individual spans, status, duration, model, provider and tool data.
• Token and cost tracking: Shows input tokens, output tokens, total tokens and the estimated cost of the model calls.
• Agents page: Provides an application-level view with the active agent, span count and accumulated cost.
• Content tracing: Can record prompts, model responses and tool data so we can investigate an incorrect result. This data may be private, so enable it only when it is safe to do so.

The SDK also provides the start and stop methods used by the application. Observability.instrument() starts monitoring before Genkit loads, and Observability.shutdown() sends the remaining telemetry before this short command exits.

These features work together. Monitoring collects the events; tracing groups them. The dashboard helps us inspect the events, and usage data helps us understand the running cost of the agent.

The main files are:

src/agent.ts                 Genkit agent, tools, and output schema
src/app.ts                   Command line application
src/tools/                   Mock flight, hotel, and activity data

We will not change the agent logic. Both runs must use the same code.

How the Pieces Fit Together

The project repository contains the complete runnable agent; we only need to change the application startup and shutdown flow:

• src/agent.ts defines the Genkit agent, its output schema, and its tools.
• src/tools/ contains the mock travel data.
• src/app.ts reads the command-line arguments, calls the agent, and prints the result.
• bootstrap.ts starts Progress, loads the application, and flushes telemetry before the process exits.

The full agent and tool code is already part of the project repository. We don’t repeat it here because observability shouldn’t require rewriting the travel logic.

Get the Project

First, clone the project repository and move into its folder. Replace <repository-url> with the URL of the repository that contains this example:

git clone https://github.com/danywalls/genkit-progress-observability.git
cd genkit-progress-observability
git branch --all

You should see the article-start and main branches. The first branch contains the agent before observability.

Before You Start

You need a Google AI Studio API key for both branches. The main branch also needs a Progress Agent Engineering integration key. Create a local .env file in the project root:

GOOGLE_API_KEY=your-google-api-key
OBSERVABILITY_API_KEY=ac_p_your-integration-key
OBSERVABILITY_APP_NAME=family-travel-planner
GEMINI_MODEL=gemini-3.5-flash-lite

The article-start branch uses only the Google key to call Gemini. The main branch uses both keys. The .env file is ignored by Git, so it will not be committed.

Run the Agent Without Observability

Start from the article-start branch:

git checkout article-start
npm install
npm start -- Ibiza 2 2 moderate 7 5

The terminal shows the travel plan and the tool messages. This is useful, but it does not answer important questions:

• Did the model call all three tools?
• Which model call took most of the time?
• How many tokens did the request use?
• How much did this request cost?
• Which step should we inspect when the answer is wrong?

The command prints a travel plan like this:

Ibiza Family Travel Planner - Genkit
Planning a family trip to Ibiza...
[tool:find-flights] Searching flights to Ibiza for 4 passengers
[tool:find-hotels] Searching hotels in Ibiza
[tool:find-activities] Finding activities in Ibiza for 2 adults and 2 kids
[agent:stream] model started producing a structured travel plan
TOTAL ESTIMATED COST: €1,350

The agent works, but it is still a black box. We need more information about the run. Now let’s add the observability bootstrap around the same application.

Add Progress Agent Engineering Observability

Add Progress Observability

If you have just run the agent from article-start, you already have the part that matters most: a working Genkit flow. We are not going to redesign that flow or add tracing calls around every tool. We are going to place a small observability layer around the application that is already there.

Here is the change we are about to make:

1. Move the command entry point to bootstrap.ts.
2. Start Progress before src/app.ts loads Genkit and the Google AI plugin.
3. Let the existing agent run as it does today.
4. Shut Progress down after the request so the last spans are sent to the dashboard.

Genkit already creates spans for the flow, model calls and tools, and Progress 3.1.1 knows how to process those Genkit spans. Its native Genkit support also maps tool names, arguments and results, as well as model token usage, into the trace. Our job is to put the start and end of the observability lifecycle in the right places. Once that is clear, the integration is easier to follow: one file controls startup, the existing application runs in the middle, and one shutdown call closes the run.

1. Install the Integration

Switch to the branch that contains the observability setup and install its dependencies:

git checkout main
npm install

The main branch changes the entry point in package.json from tsx src/app.ts to tsx bootstrap.ts. This makes bootstrap.ts the first file executed by the command. That order matters: if src/app.ts loads Genkit before Progress starts, some libraries may already be initialized and their operations may not be captured.

2. Start Progress Before Loading the Application

Create bootstrap.ts in the project root. Its job is to prepare the environment, validate the required keys, start the SDK and only then load the application:

import '@progress/observability/register/hooks';
import 'dotenv/config';

import { Observability, ObservabilityInstruments } from '@progress/observability';

const apiKey = process.env.OBSERVABILITY_API_KEY;
if (!apiKey) throw new Error('OBSERVABILITY_API_KEY is not set');

if (!process.env.GOOGLE_API_KEY) {
  throw new Error('GOOGLE_API_KEY is not set');
}

await Observability.instrument({
  appName: process.env.OBSERVABILITY_APP_NAME ?? 'family-travel-planner',
  apiKey,
  instruments: new Set([ObservabilityInstruments.GOOGLE_GENERATIVEAI]),
  traceContent: true,
});

await import('./src/app.js');

Let’s follow the file in the order Node executes it.

First, the hooks are registered:

import '@progress/observability/register/hooks';

This gives Progress a way to observe supported libraries as they are loaded. It needs to appear before the application imports Genkit or the Google AI plugin. If those libraries are loaded first, their initialization may happen before Progress has installed its instrumentation.

Next, we load the values from .env:

import 'dotenv/config';

After this import, process.env contains GOOGLE_API_KEY, OBSERVABILITY_API_KEY and OBSERVABILITY_APP_NAME. We read the Progress key and check both required keys before starting the application. Failing here gives us a clear configuration error instead of a request that cannot be traced or sent to Gemini.

Now we start the observability SDK:

await Observability.instrument({
  appName: process.env.OBSERVABILITY_APP_NAME ?? 'family-travel-planner',
  apiKey,
  instruments: new Set([ObservabilityInstruments.GOOGLE_GENERATIVEAI]),
  traceContent: true,
});

Think of instrument() as the point where we turn observability on for this process. It prepares the telemetry pipeline and configures what Progress should capture:

• appName groups this application’s traces under family-travel-planner in the dashboard.
• apiKey identifies the Progress integration that receives the telemetry.
• instruments enables the Google Generative AI instrumentation used by the Genkit Google AI plugin.
• traceContent includes prompts, responses and tool data in the trace. That helps us debug an incorrect result, but the captured content may contain sensitive information.

The call is asynchronous because the SDK needs to prepare that pipeline. await makes the bootstrap wait until the setup has completed. We do not want the agent to start while the observability layer is still initializing.

Only after instrument() finishes do we load the application:

await import('./src/app.js');

This is a dynamic import, rather than a static import at the top of the file, for one reason: it keeps Genkit and the agent from loading too early. From this point on, src/app.ts runs the same parseArgs(), planTrip() and output code as before, but its Genkit flow and model calls are now observed by Progress.

At this point, Progress can observe the Genkit flow, but a short-lived CLI process can exit before its telemetry is transmitted. We therefore need an explicit shutdown step.

3. Close the SDK After the Agent Run

Open src/app.ts. Keep the argument parsing, planTrip() call and output formatting from the first run. Add the Observability import and call shutdown() from the existing finally block:

 import { planTrip } from './agent.js';
+import { Observability } from '@progress/observability';
 
 // The existing main() function still parses arguments, calls planTrip()
 // and prints the travel plan.
 try {
   // Existing planTrip() call and output.
 } catch (error) {
   console.error('Error generating travel plan:', error);
   process.exitCode = 1;
 } finally {
+  await Observability.shutdown();
 }

finally runs after both a successful and a failed request. That makes it the correct place to flush the remaining spans. The agent logic remains unchanged: it still calls the same tools and produces the same structured travel plan. The only new behavior is that the SDK closes after the run has produced its result.

The resulting application structure is:

bootstrap.ts
  1. Load hooks and environment variables
  2. Start Progress
  3. Load src/app.ts
       4. Run the existing Genkit agent
       5. Shut down Progress in finally

This separation is useful because bootstrap.ts controls initialization while src/app.ts controls one agent run. It also explains why the application does not need custom tracing code around find-flights, find-hotels or find-activities.

At the end of the integration, the final version is organized like this:

bootstrap.ts
  - Loads the Progress hooks and environment variables
  - Validates the API keys
  - Starts Observability.instrument()
  - Loads src/app.ts only after instrumentation is ready

src/app.ts
  - Parses the command-line arguments
  - Calls the existing planTrip() function
  - Prints the travel plan
  - Calls Observability.shutdown() in finally

The important change is the execution order. Progress starts before the agent, the existing Genkit code runs without custom tracing changes, and the SDK shuts down after the request. That is the complete integration we need for this command-line application.

The application now starts Progress before loading Genkit and shuts it down after the request finishes. The agent and its tools are still unchanged. Run the same command as before:

npm start -- Ibiza 2 2 moderate 7 5

The terminal output is similar to the first run. The difference is that Progress now receives the full trace.

The command prints the travel plan and confirms that telemetry was sent:

Ibiza Family Travel Planner - Genkit + Progress Observability
Planning a family trip to Ibiza...
[agent:stream] model started producing a structured travel plan
TOTAL ESTIMATED COST: €1,350
Check your traces at: https://observability.progress.com
[Observability] Observability shutdown completed successfully

The exact travel plan and total may be different in your run. Model responses are not always identical, even when we send the same input. For this comparison, focus on the execution path and the trace data, not on matching every word or number.

Let’s read the trace in the Progress observability platform and go to Observe > Tracing. Find the family-travel-planner service.

Open the latest trace. A normal trace looks like this:

The exact number of spans can change when the model makes a different number of calls. The important point is that the trace shows the whole run in one place.

When you open the trace, read it in this order:

1. Check the trace status. Did the request finish successfully?
2. Check the total duration. Is the request slower than expected?
3. Open the tool spans. Did the agent call the expected tools, and did they return useful data? In the Tool Details panel, Progress shows the tool name, the JSON arguments sent by Genkit and the result returned by the tool.
4. Open the model spans. Which model was used, and how long did each call take?
5. Check tokens and cost. Is the request using more resources than expected?

To answer these questions, check model usage. The trace summary shows input tokens, output tokens, total tokens, input and output cost, total cost and request duration.

In one real test run, the trace showed 7 spans, 2,411 total tokens and a total cost of $0.0021. Token counts can vary between runs. The model span also shows the model name and provider information. The generate span is a Genkit flow span, so it may not have model provider data.

Finally, the Agents page gives us a wider view of the application. It shows the active agent, span count and accumulated cost.

At this point, we have followed one request from the command line to its trace. The trace connects the Genkit flow, model calls and tools, while the summary shows usage and cost and turn that data into practical debugging decisions.

Why This Helps

The second run gives us information that the first run did not provide.

If the answer is slow, we can check the duration of the model and tool spans. If the answer is wrong, we can inspect the tool output and the model input. If the cost grows, we can compare token use between runs.

This is the main value of observability. It changes a guess into information we can use, the best part is the. The project does not add custom code to create model spans or map Genkit token fields, tool names, arguments or results because Progress Observability 3.1.1 handles the Genkit data directly.

This keeps the application simple:

• Genkit runs the agent and its tools.
• Progress records the Genkit trace.
• The dashboard shows model use and cost.
• The application code stays focused on travel planning.

Conclusion

The agent worked before observability. That was enough to produce an answer, but not enough to understand it or identify the next improvement.

After adding observability, we can see the complete run: the generation steps, tools, model calls, tokens and cost. This gives us a clear place to look when the agent is slow, when the answer is wrong or when the cost is growing. We can use that information to improve the prompt, fix a tool, change the model or remove unnecessary model calls.

The integration is also small. We start the SDK before Genkit loads, run the existing agent, and shut the SDK down when the request finishes. We do not need to rewrite the agent logic or add custom code around every tool call. That makes it practical to add observability to an agent that is already running in a product.

Remember that the observability layer is not tied to one model provider. We used Genkit and Gemini, but the same idea can remain in place if the application adds another provider or changes its model. Progress gives the team one view of the request while the technology behind that request evolves.

Happy observability!!

• Source Code
• Genkit documentation
• Genkit Google AI plugin
• Progress Agent Engineering observability platform
• Progress TypeScript SDK