Kubernetes

Two OpenTelemetry Collector workloads cover an entire cluster: a DaemonSet on every node and one small Deployment for cluster-wide state. Together they give TracePath node health, pod resource usage, container logs, cluster state, and Kubernetes events, under one project token.

Both are the upstream OpenTelemetry Collector Helm chart with a TracePath values file. Nothing TracePath-specific is installed in the cluster: the images, the chart, and the receivers are all the OpenTelemetry project's, and the only TracePath ingredient is the OTLP endpoint and your project token.

What you get

The applications keep their own instrumentation. The collectors describe the cluster the applications run on; they do not replace OpenTelemetry in the app.

Nodes from every cluster and every plain host land in the same table, grouped by cluster. Machines with no k8s.cluster.name collect under Not in a Kubernetes cluster, so a half-migrated fleet still reads as one fleet.

Install

kubectl create namespace tracepath
 
kubectl create secret generic tracepath-token \
  --namespace tracepath \
  --from-literal=token='<your-project-token>'
 
kubectl create configmap tracepath-settings \
  --namespace tracepath \
  --from-literal=TRACEPATH_CLUSTER_NAME='production-eu' \
  --from-literal=TRACEPATH_ENDPOINT='https://ingest.tracepath.dev/api/otel'

mode: daemonset
 
image:
  repository: otel/opentelemetry-collector-contrib
 
presets:
  hostMetrics:
    enabled: true
  kubeletMetrics:
    enabled: true
  logsCollection:
    enabled: true
  kubernetesAttributes:
    enabled: true
 
extraEnvs:
  - name: TRACEPATH_TOKEN
    valueFrom:
      secretKeyRef:
        name: tracepath-token
        key: token
  - name: TRACEPATH_ENDPOINT
    valueFrom:
      configMapKeyRef:
        name: tracepath-settings
        key: TRACEPATH_ENDPOINT
  - name: TRACEPATH_CLUSTER_NAME
    valueFrom:
      configMapKeyRef:
        name: tracepath-settings
        key: TRACEPATH_CLUSTER_NAME
 
config:
  receivers:
    hostmetrics:
      collection_interval: 30s
      scrapers:
        cpu:
          metrics:
            system.cpu.utilization:
              enabled: true
        memory:
          metrics:
            system.memory.utilization:
              enabled: true
        filesystem:
          metrics:
            system.filesystem.utilization:
              enabled: true
    kubeletstats:
      collection_interval: 30s
      metric_groups: [node, pod]
 
  processors:
    resource/cluster:
      attributes:
        - key: k8s.cluster.name
          value: ${env:TRACEPATH_CLUSTER_NAME}
          action: upsert
        - key: k8s.node.name
          value: ${env:K8S_NODE_NAME}
          action: upsert
    resource/node:
      attributes:
        - key: service.name
          value: ${env:K8S_NODE_NAME}
          action: upsert
        - key: host.name
          value: ${env:K8S_NODE_NAME}
          action: upsert
    transform/log_service_name:
      error_mode: ignore
      log_statements:
        - context: resource
          statements:
            - set(attributes["service.name"], attributes["k8s.container.name"]) where attributes["service.name"] == nil
 
  exporters:
    otlphttp/tracepath:
      endpoint: ${env:TRACEPATH_ENDPOINT}
      headers:
        Authorization: Bearer ${env:TRACEPATH_TOKEN}
      compression: gzip
      retry_on_failure:
        enabled: true
      sending_queue:
        enabled: true
        queue_size: 1000
 
  service:
    pipelines:
      traces: null
      metrics:
        receivers: [hostmetrics, kubeletstats]
        processors:
          [memory_limiter, k8sattributes, resource/cluster, resource/node, batch]
        exporters: [otlphttp/tracepath]
      logs:
        receivers: [filelog]
        processors:
          [
            memory_limiter,
            k8sattributes,
            resource/cluster,
            transform/log_service_name,
            batch,
          ]
        exporters: [otlphttp/tracepath]
 
resources:
  requests:
    cpu: 100m
    memory: 200Mi
  limits:
    memory: 500Mi

mode: deployment
replicaCount: 1
 
image:
  repository: otel/opentelemetry-collector-contrib
 
presets:
  clusterMetrics:
    enabled: true
  kubernetesEvents:
    enabled: true
 
extraEnvs:
  - name: TRACEPATH_TOKEN
    valueFrom:
      secretKeyRef:
        name: tracepath-token
        key: token
  - name: TRACEPATH_ENDPOINT
    valueFrom:
      configMapKeyRef:
        name: tracepath-settings
        key: TRACEPATH_ENDPOINT
  - name: TRACEPATH_CLUSTER_NAME
    valueFrom:
      configMapKeyRef:
        name: tracepath-settings
        key: TRACEPATH_CLUSTER_NAME
 
config:
  receivers:
    k8s_cluster:
      collection_interval: 30s
      node_conditions_to_report: [Ready, MemoryPressure, DiskPressure, PIDPressure]
      allocatable_types_to_report: [cpu, memory, storage, ephemeral-storage]
 
  processors:
    resource/cluster:
      attributes:
        - key: k8s.cluster.name
          value: ${env:TRACEPATH_CLUSTER_NAME}
          action: upsert
        # Cluster-wide series belong to no node, so they report under the
        # cluster's own name rather than any instance.
        - key: service.name
          value: ${env:TRACEPATH_CLUSTER_NAME}
          action: upsert
 
  exporters:
    otlphttp/tracepath:
      endpoint: ${env:TRACEPATH_ENDPOINT}
      headers:
        Authorization: Bearer ${env:TRACEPATH_TOKEN}
      compression: gzip
      retry_on_failure:
        enabled: true
 
  service:
    pipelines:
      traces: null
      metrics:
        receivers: [k8s_cluster]
        processors: [memory_limiter, resource/cluster, batch]
        exporters: [otlphttp/tracepath]
      logs:
        receivers: [k8sobjects]
        processors: [memory_limiter, resource/cluster, batch]
        exporters: [otlphttp/tracepath]
 
resources:
  requests:
    cpu: 100m
    memory: 200Mi
  limits:
    memory: 500Mi

helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm repo update
 
helm upgrade --install tracepath-node-agent \
  open-telemetry/opentelemetry-collector \
  --namespace tracepath \
  --values node-agent-values.yaml
 
helm upgrade --install tracepath-cluster-agent \
  open-telemetry/opentelemetry-collector \
  --namespace tracepath \
  --values cluster-agent-values.yaml

kubectl -n tracepath get daemonset,deployment
kubectl -n tracepath get pods -l app.kubernetes.io/instance=tracepath-node-agent
kubectl -n tracepath logs -l app.kubernetes.io/instance=tracepath-node-agent --tail=50

Naming: what becomes an instance

TracePath's identity for a reporting machine is the server_name tag, and server_name comes from the service.name resource attribute. Nothing else creates or names an instance.

The node agent therefore sets service.name to the node name:

resource/node:
  attributes:
    - key: service.name
      value: ${env:K8S_NODE_NAME}
      action: upsert

K8S_NODE_NAME is not something you declare: the chart injects it into every collector pod from the downward API (spec.nodeName), which is what makes the same values file resolve to a different name on each node. One node, one service.name, one instance row. Skip the processor and every node in the cluster reports under the collector's own service name and collapses into a single row.

The same rule is why applications must keep their own service.name. A pod exporting as checkout-api stays checkout-api in Endpoints, Issues and Tasks, whatever node it lands on. Pods are not instances in the overview; nodes are.

Which Kubernetes attributes survive

TracePath does not copy arbitrary resource attributes onto stored telemetry, because every distinct value multiplies the number of series it keeps. What it keeps depends on the signal:

That shapes how you instrument:

• Group metric widgets by k8s.pod.name, k8s.node.name, k8s.namespace.name or k8s.deployment.name. All four are available as tags.
• Filter logs by any Kubernetes attribute; they are all in the record.
• Do not expect pod names on spans. If a trace needs to carry which pod served it, set it as a span attribute in the application, not as a resource attribute. service.name is what distinguishes services on the trace side.

Application telemetry

Applications in the cluster export OpenTelemetry as they normally would. Two ways to wire it:

kubectl create secret generic tracepath-otlp -n checkout \
  --from-literal=headers='Authorization=Bearer <your-project-token>'

env:
  - name: OTEL_EXPORTER_OTLP_ENDPOINT
    value: https://ingest.tracepath.dev/api/otel
  - name: OTEL_EXPORTER_OTLP_HEADERS
    valueFrom:
      secretKeyRef:
        name: tracepath-otlp
        key: headers
  - name: OTEL_SERVICE_NAME
    value: checkout-api

TracePath's ingest URL is the same OTLP/HTTP endpoint documented in the OpenTelemetry guide; nothing about it is Kubernetes-specific.

Dashboards

Open the command palette with ⌘K and install the OTelemetry Server Agent template. It is already wired to the hostmetrics names the node agent reports, so it works unchanged for nodes.

For pod-level views, build widgets on the kubelet metrics and group by a Kubernetes tag: k8s.pod.memory.usage grouped by k8s.namespace.name, the pod CPU metric grouped by k8s.pod.name, and so on. Grouping by tag is described in Dashboards.

Take the exact metric names from the widget editor's dropdown rather than from here. It lists what your project has actually received, and the kubeletstats receiver has renamed metrics across Collector versions — pod CPU is k8s.pod.cpu.usage on current builds and k8s.pod.cpu.utilization on older ones, so the name that works depends on the image you pinned.

Dashboards belong to the organization, so one Kubernetes dashboard applies to every project that reports a cluster.

Cardinality

Kubernetes generates a lot of distinct values, and each combination of tag values is a stored series.

• metric_groups: [node, pod] is what the node-agent values above set. Adding container multiplies pod series by the number of containers per pod.
• Pod names change on every rollout. A deployment that ships ten times a day leaves ten times as many k8s.pod.name values behind in a week. Group dashboards by k8s.deployment.name for anything you want stable over time, and reach for k8s.pod.name when debugging a specific pod.
• Wide time ranges read pre-aggregated rollups rather than raw points, so cardinality costs storage more than it costs query speed. See Metrics.

Logs

The chart's logsCollection preset tails /var/log/pods/*/*/*.log through the filelog receiver and uses the collector's container operator, so containerd, CRI-O and Docker formats all parse, and k8s.pod.name, k8s.namespace.name and k8s.container.name come off the file path.

Pod logs have no service.name of their own, and the Logs page filters by service, so the values file falls back to the container name:

transform/log_service_name:
  error_mode: ignore
  log_statements:
    - context: resource
      statements:
        - set(attributes["service.name"], attributes["k8s.container.name"]) where attributes["service.name"] == nil

Two things worth knowing:

• The preset starts at the end of each file, so it does not replay history on first install. A checkpoint file on each node keeps its position across restarts.
• The preset excludes the collector's own pods from log collection. Without that, the collector's output becomes logs, which become collector output.

Application logs that already carry a trace id join up with their trace on the Logs page. Container stdout does not, unless the application writes the trace id into the line. Exporting logs through OpenTelemetry from inside the app is what gives you that linkage; see OTel Logs.

Alerting on the cluster

Cluster telemetry feeds the same alerting and on-call machinery as everything else. Useful rules:

• A metric rule on k8s.container.restarts, to page on a crash-looping deployment before anyone notices the 500s.
• A metric rule on system.memory.utilization per node, ahead of the eviction threshold.
• A Monitor probing an Ingress hostname, so you find out when the cluster is reachable but the application is not.

Troubleshooting

The Collector image is built FROM scratch: it contains the binary and nothing else, so kubectl exec into it has no shell, no grep and no printenv. Everything below inspects the objects and the logs instead, which is why none of these commands enter the container.

Start with what is running and what it is saying:

kubectl -n tracepath get daemonset,deployment
kubectl -n tracepath logs -l app.kubernetes.io/instance=tracepath-node-agent --tail=50

No instances in the overview. The rows come from system.cpu.utilization. The chart renders the merged config into a ConfigMap, so read it from there:

kubectl -n tracepath get configmap \
  -l app.kubernetes.io/instance=tracepath-node-agent \
  -o jsonpath='{.items[0].data.relay}'

Look for system.cpu.utilization under the cpu scraper with enabled: true. If the key is absent, your values file did not merge the way you expected.

One instance instead of one per node. service.name is not resolving per node. It is set from K8S_NODE_NAME, which the chart injects from the downward API (spec.nodeName) — you do not declare it yourself, but a values file that overrides the container's env wholesale can drop it. Confirm the DaemonSet still carries it, and that there is one pod per node:

kubectl -n tracepath get daemonset -l app.kubernetes.io/instance=tracepath-node-agent \
  -o jsonpath='{.items[0].spec.template.spec.containers[0].env[?(@.name=="K8S_NODE_NAME")]}'
 
kubectl -n tracepath get pods -l app.kubernetes.io/instance=tracepath-node-agent \
  -o custom-columns=POD:.metadata.name,NODE:.spec.nodeName

The node column is what each pod reports as its service.name, so it is also the list of instance names you should see in the overview.

Instances are not grouped by cluster. The Group by Kubernetes cluster option only appears once at least one instance carries k8s.cluster.name. Check TRACEPATH_CLUSTER_NAME is set in the tracepath-settings ConfigMap and the pods were restarted after it changed — a ConfigMap change does not restart them on its own.

CPU shows but memory and disk are blank. system.memory.utilization and system.filesystem.utilization are separately opt-in and are easy to miss when adapting an existing collector config.

401 from the exporter. The collector logs the status code. The token must be the project token from the project's Connection page, not a personal access token or a source-map upload token.

503 with a Retry-After header. Either the ingest tier is shedding a burst (Retry-After: 2), or the organization has used its monthly ingest allowance or is suspended (Retry-After: 60); the response body says which. The collector retries on its own while retry_on_failure is enabled, which clears the first but not the other two — see When ingest answers 503.

413 from the exporter. A single export exceeded the 10 MB decompressed body limit. Lower the batch processor's send_batch_size.

No pod logs. Confirm /var/log/pods is mounted and readable. On a node with a read-only root filesystem the checkpoint hostPath may also fail to create; the collector logs that at startup.

Exporter looks healthy but nothing appears. Check the project, not the collector. Open Dashboards, add a widget, and search the metric dropdown for system.cpu.utilization: if the name is not there, the points are landing in a different project than the one you are looking at.

Next steps

• Organization Overview for what the Servers page does with all this
• OpenTelemetry for instrumenting the applications
• Resource Attributes for the full allowlist
• Dashboards for building pod and namespace views