OpenSearch Integration

Updated on Oct 5, 2026

Forge can upload transaction events, trace files, and system logs to a self-hosted OpenSearch cluster. Use it to analyze workflow runs, token usage, and failures across repositories and developers. When OpenSearch is not configured, Forge logs locally only.

Note: To send telemetry to an OpenTelemetry (OTLP) backend instead, such as Progress Observability, see Progress Observability Integration.

What Forge uploads

Data	Index
Transaction events	forge-transactions-<date> (one index per day)
Trace files (composed prompt and agent transcript)	forge-traces
System logs	forge-system-logs-<date>

These forge-* names apply to current Forge releases. Older clients may use different index prefixes; apply the index templates and role grants that match the client version you are running.

Prerequisites

• An OpenSearch cluster with the Forge index templates applied. See Set up the cluster.
• An OpenSearch user or API token with write access to the forge-* indices.

Configure Forge

Add an [opensearch] section to telemetry.toml:

toml

[opensearch]
uri = "https://opensearch.example.com:9200"
api_token = "your-api-token"
accept_invalid_certs = false
enforce = false

Key	Description	Default
uri	OpenSearch endpoint URL. Required.	—
api_token	Bearer token, or Base64-encoded username:password for Basic authentication. Required.	—
accept_invalid_certs	Accept self-signed or invalid TLS certificates. Use only on trusted internal networks.	false
enforce	Stop Forge when no valid OpenSearch configuration is found.	false

To create a Basic authentication token:

bash

echo -n 'user:password' | base64

Forge reads telemetry.toml from these locations:

Scope	Path
Project	.forge/config/telemetry.toml
User	~/.config/forge/telemetry.toml (or $XDG_CONFIG_HOME/forge/telemetry.toml)
Organization	/etc/forge/telemetry.toml
Override	The path in FORGE_TELEMETRY_PATH

You can also keep the [opensearch] section in a separate opensearch.toml file, for example .forge/config/opensearch.toml or ~/.config/forge/opensearch.toml. frg config init lists both .forge/config/opensearch.toml and .forge/config/telemetry.toml in its recommended .gitignore patterns. Configure OpenSearch in one file only.

Note: Forge has no FORGE_OPENSEARCH_URI, FORGE_OPENSEARCH_API_TOKEN, or FORGE_OPENSEARCH_ACCEPT_INVALID_CERTS environment variables. In CI, write telemetry.toml from a secret and set FORGE_TELEMETRY_PATH. FORGE_OPENSEARCH_BATCH_SIZE and FORGE_OPENSEARCH_FLUSH_INTERVAL_SECS are deprecated and ignored.

Protect the credentials

Keep credential files out of version control and readable only by you:

bash

chmod 600 ~/.config/forge/telemetry.toml

See Version Control Setup for the recommended .gitignore patterns.

Require OpenSearch

To make OpenSearch mandatory, set enforce = true. An organization can provide the URI in /etc/forge/telemetry.toml and let each developer supply api_token in their user file:

toml

[opensearch]
uri = "https://company-opensearch.internal:9200"
api_token = ""
enforce = true

With enforce = true, Forge exits with an error when no source provides a valid configuration.

How uploads work

• Forge writes each event to the local transaction log first, then uploads it to OpenSearch synchronously, one request per event. Uploads are not batched.
• Upload failures are logged as warnings and never fail the workflow. The local log is authoritative.
• Each upload adds latency. A full frg workflow run emits about 100 events, so the total can reach tens of seconds even when OpenSearch is healthy.
• A circuit breaker stops uploads after repeated failures and retries later. While it is open, uploads are skipped without network delay.
• Trace files that are too large are skipped.
• When telemetry is disabled, OpenSearch uploads are disabled too.

Data schema

Transaction events

Field	Type	Description
@timestamp	date	Event timestamp (not upload time)
event_type	keyword	Event type (see Event types)
job_id	keyword	Unique job identifier
command	keyword	Workflow command executed
repository	keyword	Repository name
repository_owner	keyword	Repository owner
repository_remote	keyword	Git remote URL (sanitized)
user_name	keyword	Git user.name
user_email	keyword	Git user.email
instance_id	keyword	Globally unique id of the execution that emitted this document (<type>-<uuid>)
callers	keyword	Ancestor instance ids, root first. Absent for a root execution; callers[0] is the root, callers.last() is the direct parent, callers.length is the depth
start_time	date	Workflow start timestamp
end_time	date	Workflow end timestamp
success	boolean	Whether workflow succeeded
trace_file	keyword	Path to trace file (for linking)
model	keyword	AI model used (e.g., claude-sonnet-5). Value is "not set" for start events
role	keyword	Role prompt used (e.g., product_manager). Value is "none" when custom agent is used
custom_agent	keyword	Custom agent name (e.g., python-expert). Value is "none" when not configured
agent	keyword	AI agent platform (e.g., copilot, opencode)
token_usage.input_tokens	long	AI input tokens consumed
token_usage.cached_tokens	long	AI cached tokens (cache reads, not billed)
token_usage.cache_write_tokens	long	AI cache write tokens (cache creation, priced separately; omitted when not reported)
token_usage.output_tokens	long	AI output tokens consumed
workflow_type	keyword	Workflow source: built-in, modified, or custom
role_prompt_type	keyword	Role prompt source: built-in, custom, or disabled when no role prompt was used
task_prompt_type	keyword	Task prompt source: built-in or custom
workflow_id	keyword	Work item the workflow is running against
gate_id	keyword	Approval gate identifier (approval events)
step	keyword	Workflow state the approval gate belongs to (approval events)
approver_email	keyword	Email of the approver (approval events). Hashed when [privacy] strict_privacy = true
approval_code	keyword	Approval code used to release the gate (approval events)
approval_method	keyword	manual or auto_bypassed (approval events)
parameters	object	Event-specific payload (state names, step/check ids, durations, conditions)
context	object	Job context captured on the first transition (job dir, issue/PR/ticket id)
invocation_source	(dynamic)	manual or flow, derived from callers. No explicit mapping is defined for this field — it exists for the App Insights sink, not for OpenSearch aggregation

model, role, and custom_agent always contain a string. Exclude the sentinel values "not set" and "none" when you aggregate.

workflow_type is omitted when an execution does not resolve a single catalog command. role_prompt_type is omitted when no role prompt was composed; disabled explicitly records a composed prompt with no role. task_prompt_type is omitted when no task prompt was composed. Auto-retry continuations retain workflow_type and omit prompt-provenance fields because the retry prompt is generated at runtime. These keys are absent rather than null or defaulted, so queries and dashboards should tolerate missing fields. After upgrading to a build that sends these fields, refresh the forge-transactions-* index pattern so it picks up the mappings. Existing indexed documents are not backfilled.

To query call chains, use callers, not invocation_source. callers[0] is the root and the array length is the depth. Orchestrating commands such as frg workflow run, frg app <target> <op>, and frg learn run appear in callers but emit no document of their own, so the root ID often has no matching instance_id.

Event types

event_type	Emitted by	Notes
workflow	AI workflow commands	Command start and completion
utility	Utility commands (config, status, …)	No job context
config_change	Configuration writes	Old and new values
validation	Input validation	Context file counts
output_tracking	Output validation	Expected and produced files
auto_retry	Auto-retry	Missing output count
workflow_started	Workflow engine	One per frg workflow run
workflow_state_transition	Workflow engine	One per state change
branch_evaluated	Workflow engine	Condition result and target state
step_execution	Workflow engine, command hooks	Shell, builtin, and agent steps
check_evaluation	Workflow engine, command hooks	Check result and action
approval	Workflow engine	Carries approver_email and approval_code

Workflow engine events are uploaded without sampling. A typical frg workflow run issue-to-pr produces about 86 documents (about 30 KB) instead of about 26.

Warning: approval documents contain approver_email and approval_code. They reach your OpenSearch cluster only; OTLP spans and Progress Analytics do not receive them. Set [privacy] strict_privacy = true in telemetry.toml to hash repository, git user, and approver identity before upload. See Security Guide.

Trace documents

Field	Type	Description
@timestamp	date	Trace creation time
job_id	keyword	Job identifier
trace_file_path	keyword	Matches trace_file in transaction events
command	keyword	Command
repository, repository_owner	keyword	Repository name and owner
user_name, user_email	keyword	User identity
trace_content	text	Full trace file content

To find the trace for a job, read trace_file from its transaction event, then query forge-traces:

json

GET forge-traces/_search
{
 "query": { "term": { "trace_file_path": "traces/20260421_102030_workflow.trace.md" } }
}

Set up the cluster

Progress provides helper scripts that create the Forge role and user, apply the index templates for transactions, traces, and system logs, and import dashboards. They are not publicly distributed. Contact your Progress representative for the scripts and index templates.

If you use the dashboards, they are imported to the Global tenant. In OpenSearch Dashboards, select your user name > Switch tenants > Global, then open Dashboards. To build your own visualizations, use the index patterns forge-transactions-*, forge-traces, and forge-system-logs-*.

To verify the setup, run an AI workflow command, then check that documents arrive:

bash

curl -H "Authorization: Bearer $OPENSEARCH_TOKEN" \
 "https://opensearch.example.com:9200/forge-transactions-*/_count"

Example queries

Token usage by user:

json

GET forge-transactions-*/_search
{
 "size": 0,
 "query": { "term": { "event_type": "workflow" } },
 "aggs": {
 "by_user": {
 "terms": { "field": "user_email" },
 "aggs": {
 "total_input": { "sum": { "field": "token_usage.input_tokens" } },
 "total_output": { "sum": { "field": "token_usage.output_tokens" } }
 }
 }
 }
}

Failed workflows in the last 7 days:

json

GET forge-transactions-*/_search
{
 "query": {
 "bool": {
 "must": [
 { "term": { "event_type": "workflow" } },
 { "term": { "success": false } },
 { "range": { "@timestamp": { "gte": "now-7d" } } }
 ]
 }
 }
}

Token usage by model, excluding sentinel values:

json

GET forge-transactions-*/_search
{
 "size": 0,
 "query": {
 "bool": {
 "must": [ { "term": { "event_type": "workflow" } }, { "exists": { "field": "token_usage" } } ],
 "must_not": [ { "term": { "model": "not set" } } ]
 }
 },
 "aggs": {
 "by_model": {
 "terms": { "field": "model", "size": 10 },
 "aggs": {
 "total_input": { "sum": { "field": "token_usage.input_tokens" } },
 "total_output": { "sum": { "field": "token_usage.output_tokens" } }
 }
 }
 }
}

Workflows that use customizations:

json

GET forge-transactions-*/_search
{
 "query": {
 "bool": {
 "should": [
 { "term": { "workflow_type": "custom" } },
 { "term": { "role_prompt_type": "custom" } },
 { "term": { "task_prompt_type": "custom" } }
 ],
 "minimum_should_match": 1
 }
 }
}

Security considerations

• Trace documents contain prompts and agent transcripts, which can include code. Restrict access with OpenSearch role-based access control.
• Use HTTPS. Set accept_invalid_certs = true only for trusted internal clusters.
• Forge removes embedded credentials from git remote URLs before upload.
• Consider a VPN or private network between developers and the cluster.

Troubleshooting

No data appears in OpenSearch

Cause: Forge did not load a valid [opensearch] section, the index templates are missing, or uploads fail.

Solution:

1. Check the file Forge reads, and that uri and api_token are set.
2. Run an AI workflow command with RUST_LOG=debug and search the logs:

bash

grep -i opensearch .forge/work/*/logs/system.log

Look for OpenSearch integration enabled with synchronous uploads, OpenSearch not configured - uploads disabled, or Telemetry disabled - OpenSearch uploads disabled. 3. Check that the index templates exist on the cluster.

Invalid OpenSearch URI format in telemetry.toml

Cause: uri is not a valid URL.

Solution: Use a full URL with scheme and port, for example https://opensearch.example.com:9200.

OpenSearch API token is empty in telemetry.toml

Solution: Set api_token in [opensearch].

OpenSearch upload skipped - circuit breaker open

Cause: Several uploads failed in a row. Workflows continue; uploads resume automatically.

Solution:

1. Check cluster health and network connectivity.
2. Check the token and its index permissions.
3. Review system.log for the first failure.

Certificate validation errors

Cause: The cluster uses a self-signed or untrusted certificate.

Solution: Fix the certificate chain. For trusted internal clusters only, set accept_invalid_certs = true.

Limitations

• Delivery is best effort and failed uploads are not retried within a workflow.
• Uploads are synchronous and add latency per event.
• Oversized trace files are skipped.
• Index lifecycle management is not included; configure it on the cluster.

User identity

Forge attaches a user identity to uploaded events. It resolves the identity in this order:

1. FORGE_USER_NAME and FORGE_USER_EMAIL environment variables
2. User settings in .forge/context.toml
3. git config user.name and user.email
4. Operating system user
5. The fallback value unresolved

Set the identity for a project, for example in containers without git config:

bash

frg config user --name "Jane Doe" --email "[e-mail]"

The identity persists across frg config clear-context. Run frg config show-context to see the resolved identity.

In CI, set the environment variables from your platform's user variables:

yaml

# GitHub Actions
env:
 FORGE_USER_NAME: ${{ github.actor }}
 FORGE_USER_EMAIL: ${{ github.actor }}@users.noreply.github.com

Platform	FORGE_USER_NAME	FORGE_USER_EMAIL
GitLab CI	$GITLAB_USER_NAME	$GITLAB_USER_EMAIL
Azure Pipelines	$(Build.RequestedFor)	$(Build.RequestedForEmail)
Jenkins	${env.GIT_COMMITTER_NAME}	${env.GIT_COMMITTER_EMAIL}

To see which source Forge used, run:

bash

RUST_LOG=debug frg config show-context 2>&1 | grep "Resolved user"

The log shows env_var, context_toml, git_config, or system.

Note: User name and email reach your OpenSearch cluster unless [privacy] strict_privacy = true. Contact your Forge administrator for your organization's retention policy.

See Also

• Progress Observability Integration
• Transaction Log Format
• Security Guide
• Environment Variables
• Version Control Setup