Traces

TracePath ingests OpenTelemetry spans via the OTLP/HTTP protocol and maps them to its own data model.

Span Mapping

Spans are classified by kind and attributes, not by whether they're a root span. Both root and child spans can be promoted to a top-level concept when the signal is there. A queued job processed as a child of the dispatcher's trace still becomes a Task, and an LLM call made inside a request handler still becomes an AI Trace.

The rules below are checked in order, and the first match wins.

Order matters most for gen_ai.*, which is checked last. A CONSUMER span carrying gen_ai.* becomes a Task, and a request span carrying gen_ai.* becomes an Endpoint. Neither shows up on AI Traces. Put every model call on its own child span so rule 5 is the one that matches it.

The default span kind is INTERNAL, and a root INTERNAL span with no HTTP attributes is discarded. If you wrap a cron job or a worker in tracer.startActiveSpan("my-job") without setting a kind, nothing appears on Tasks and nothing appears on Spans. TracePath still answers 200 OK. Set { kind: SpanKind.CONSUMER } for queue and job work, or add a console.command attribute for CLI commands. The same applies to root CLIENT and PRODUCER spans, and to root SERVER spans that carry no HTTP attributes.

"HTTP attributes" means at least one of http.request.method, http.method, http.route or url.path. Nothing else counts, so a span carrying only url.full, http.target or server.address is not treated as an HTTP entry point.

Rule 1 exists because browser SDKs stamp page context like url.path onto their error spans, and those must become Issues rather than endpoints. It applies to exception.* set as span attributes, not to a recorded exception event, and it suppresses every promotion, so an INTERNAL span with both exception.* and gen_ai.* attributes yields an Issue and no AI Trace. To record an error on a span you also want promoted, use recordException (which writes an event) instead of setting the attributes yourself.

Child spans that carry db.query.text (or the legacy db.statement) are displayed under the query text instead of the span name, so a database waterfall reads as the statements that ran rather than a column of identical pg.query rows.

Each Endpoint / Task / AI Trace row carries an is_root flag. Rows triggered by another trace surface a Non Root chip in the dashboard list, or a Mixed chip when a grouped row holds both root and non-root runs. The detail page renders a Distributed Trace card with the full waterfall (endpoint → producer → consumer, request → child LLM call) whenever the trace touches more than one run. The distributed_trace_id that links the rows is derived from the OTel trace_id, so cross-service lookup works with no vendor attribute. If you need to stitch runs that genuinely carry different trace ids, set the span attribute tracepath.distributed_trace_id to a shared UUID and it overrides the derived value.

Sub-handler spans are not duplicated, as long as parent and child ship together. An in-process child SERVER/INTERNAL+HTTP span (a framework's "handler /path" sub-span) stays in the Spans waterfall when its parent arrives in the same ResourceSpans block. Parents are matched inside one block, not across the whole export, though spans from one service normally share a Resource and land in the same block. TracePath cannot see process boundaries, so it treats "parent missing from this block" as a cross-service hop and promotes the child to its own Endpoint. If you see the same route listed twice with a Mixed chip, a batch processor split the parent and the child across two exports. Raise the exporter's maxExportBatchSize and scheduledDelayMillis so a whole trace ships together, or stop emitting the redundant sub-handler span.

Resource Attributes

These resource-level attributes are extracted and applied to all spans in the resource:

HTTP Semantic Conventions

For root SERVER spans with HTTP attributes, TracePath extracts the following:

The endpoint name is built as "METHOD /route", for example "GET /api/users".

A method attribute is required for the route to be used at all. If neither http.request.method nor http.method is set, the raw span name becomes the endpoint name and url.path is ignored entirely.

Prefer http.route (the low-cardinality template, /users/:id) over url.path (the concrete URL, /users/8412). Falling back to url.path creates one endpoint row per distinct URL.

Unmatched 404s

A 404 response whose span carries no real route is stored under the single endpoint name UNMATCHED instead of its URL. "No real route" means http.route is missing, does not start with /, or is a catch-all made only of slashes and wildcards such as /, /* or /**, which is what not-found handlers usually report.

This keeps scanner traffic and typo'd URLs from creating one endpoint row per bogus path. A 404 returned from a concretely matched route, for example GET /users/:id for a user that does not exist, keeps its own identity. Only status 404 collapses. Other statuses on a catch-all route keep the catch-all name.

Streaming Responses (SSE / WebSocket)

Long-lived streaming routes are flagged as is_stream on ingest so that the dashboard excludes them from P50/P95/P99, Apdex, and impact scoring. They still appear in the endpoints list with their count, status code, error rate, and throughput. Detection signals (any of):

There is no OpenTelemetry-standard streaming attribute, so TracePath sniffs the captured Content-Type response header. Two things trip people up. The attribute key must be exactly lowercase, http.response.header.content-type, which is what the OTel HTTP conventions emit. And the match is a prefix, so a comma-joined value like application/json, text/event-stream is not detected.

Many SDKs require explicit opt-in before response headers are captured at all. When in doubt, set tracepath.is_stream = true on the span yourself. It must be a boolean. The string "true" is ignored.

Exception Events

Spans with events named "exception" are extracted as Issues in TracePath. The following event attributes are used:

Exceptions are grouped by a normalized hash of the stack trace, so the same logical error produces the same Issue regardless of runtime differences like memory addresses or user IDs.

TracePath reads exception data from two places: an event named exception on the span, which is what span.recordException(err) produces, and exception.type / exception.message / exception.stacktrace set directly as span attributes. The event wins when both are present.

The Issue is attributed to the nearest enclosing Endpoint, Task or AI Trace, walking up the parent chain from the span that raised it. An error thrown in a database span therefore shows up on the request that ran the query.

An exception on a dropped span still becomes an Issue, but an unlinked one. If you record an error on a root SpanKind.INTERNAL span, the span itself is discarded (see Span Mapping) while the Issue is still created. There is then no Endpoint or Task for it to point at, so the occurrence carries only the raw trace id and you lose the request or job context that would tell you what was running. Record exceptions on spans that sit inside a promoted entity, or fix the span kind.

Example: Node.js with OTel SDK

npm install @opentelemetry/sdk-node @opentelemetry/auto-instrumentations-node \
  @opentelemetry/instrumentation @opentelemetry/exporter-trace-otlp-http \
  @opentelemetry/resources

// telemetry.mjs
// Load it before your app: node --import ./telemetry.mjs server.js
import { register } from "node:module";
register("@opentelemetry/instrumentation/hook.mjs", import.meta.url);
 
import { NodeSDK } from "@opentelemetry/sdk-node";
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http";
import { getNodeAutoInstrumentations } from "@opentelemetry/auto-instrumentations-node";
import { resourceFromAttributes } from "@opentelemetry/resources";
 
const sdk = new NodeSDK({
  resource: resourceFromAttributes({
    "service.name": "my-service",
    "service.version": "1.0.0",
  }),
  traceExporter: new OTLPTraceExporter({
    url: "https://ingest.tracepath.dev/api/otel/v1/traces",
    headers: {
      Authorization: "Bearer your-project-token",
    },
  }),
  instrumentations: [getNodeAutoInstrumentations()],
});
 
sdk.start();

The two node:module lines are required for an ESM app (.mjs, or "type": "module"). Without the loader hook nothing is patched, so http.route is never set and your endpoints are named after raw span names. A CommonJS app does not need them. Use resourceFromAttributes, not the old new Resource(...), which was removed in @opentelemetry/resources 2.x.

This automatically instruments HTTP servers, database clients, and other libraries. SERVER spans become Endpoints, CONSUMER spans become Tasks, a span whose only signal is gen_ai.* becomes an AI Trace, and exception events become Issues.

This snippet exports traces only. Metrics and logs each need their own exporter. See the full three-signal setup on the overview page.

Example: Python (zero-code)

Most Python apps need no instrumentation code at all. opentelemetry-instrument starts the SDK, patches your web framework, and flushes on exit:

pip install opentelemetry-distro opentelemetry-exporter-otlp
opentelemetry-bootstrap -a install

export OTEL_SERVICE_NAME=my-python-service
export OTEL_EXPORTER_OTLP_ENDPOINT=https://ingest.tracepath.dev/api/otel
export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer your-project-token"
export OTEL_TRACES_EXPORTER=otlp
 
opentelemetry-instrument uvicorn app:app --port 8000      # FastAPI, any ASGI app
opentelemetry-instrument flask --app wsgi run --port 8000 # Flask

OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf is required. Left unset, the Python SDK resolves otlp to gRPC, which TracePath does not accept, and the app runs and exits with no error while nothing arrives.

The FastAPI, Flask, Starlette, and Django instrumentations set http.route from the matched route pattern in the framework's own syntax (/users/{user_id} on FastAPI, /orders/<order_id> on Flask), so endpoints arrive grouped. The Python guide covers logs, metrics, background tasks, and exceptions on this same path.

Example: Python with OTel SDK

To wire the SDK by hand instead, build the provider and create spans with a kind TracePath keeps. A provider on its own records nothing. The SDK registers an atexit hook, so a clean exit flushes the last batch even without the explicit provider.shutdown() below; only an abrupt end such as os._exit or SIGKILL loses it.

pip install opentelemetry-sdk opentelemetry-exporter-otlp-proto-http

from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.trace import SpanKind, Status, StatusCode
 
resource = Resource.create({
    "service.name": "my-python-service",
    "service.version": "1.0.0",
})
 
provider = TracerProvider(resource=resource)
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter(
    endpoint="https://ingest.tracepath.dev/api/otel/v1/traces",
    headers={"Authorization": "Bearer your-project-token"},
)))
trace.set_tracer_provider(provider)
 
tracer = trace.get_tracer("my-app")
 
 
def do_work():
    ...  # the job's real work
 
 
with tracer.start_as_current_span(
    "cleanup-expired-sessions", kind=SpanKind.CONSUMER
) as span:
    try:
        with tracer.start_as_current_span("delete-batch"):
            do_work()
        span.set_status(Status(StatusCode.OK))
    except Exception as error:
        span.record_exception(error)
        span.set_status(Status(StatusCode.ERROR, str(error)))
        raise
 
provider.shutdown()

That run appears under Tasks as cleanup-expired-sessions, with delete-batch as a child span. In Python the span kind is a keyword argument, kind=SpanKind.CONSUMER. Drop it and the root span defaults to INTERNAL, which is discarded on ingest.

Python's Resource.create(...) is unaffected by the JavaScript Resource removal. Only the JS SDK changed.

Next Steps

• Metrics: export metrics to TracePath via OTel
• Logs: export logs and link them to these traces
• Overview: endpoint, authentication, limits, quota, and a "nothing is showing up" checklist