Distributed Tracing: Jaeger Implementation in Practice

Why Distributed Tracing

In a microservices architecture, a single user request often traverses multiple services. When an endpoint’s latency spikes from 200ms to 2s, logs are scattered across N machines, making it difficult to pinpoint the bottleneck — is it slow gateway forwarding, a slow downstream DB query, or queuing in an inter-service call?

Distributed tracing solves exactly this problem: it generates a globally unique Trace ID for each request, propagates it across services, and ultimately renders a complete call chain tree in the UI, making the duration of each segment visible at a glance.

Jaeger (pronounced roughly “yay-ger”) was open-sourced by Uber and is now a CNCF graduated project. This article is based on Jaeger v1.60+ and the OpenTelemetry SDK. Official Documentation

Core Concepts of Distributed Tracing

Trace

A Trace represents a complete distributed request chain, identified by a unique 128-bit Trace ID. It is a tree structure composed of multiple Spans:

Trace (TraceID: a1b2c3...)
├── Span: HTTP GET /api/orders         [gateway]
│   ├── Span: RPC GetUser             [user-service]
│   │   └── Span: SELECT * FROM users [mysql]
│   └── Span: RPC GetOrderList         [order-service]
│       └── Span: Redis GET           [redis]

Span

A Span is the smallest unit of tracing, recording the start and end of an operation. Key fields:

Field	Description
`TraceID`	Globally unique ID of the parent Trace
`SpanID`	Unique ID of the current Span
`ParentSpanID`	Parent Span ID, used to build the call tree
`OperationName`	Operation name, e.g., `GET /api/orders`
`StartTime` / `Duration`	Start time and duration
`Tags`	Structured tags, e.g., `http.status_code=200`
`Logs`	Timestamped events, e.g., exception stack traces
`SpanKind`	CLIENT / SERVER / PRODUCER / CONSUMER / INTERNAL

Context Propagation

Context propagation is the foundation of distributed tracing. When Service A calls Service B, it must pass the TraceID, SpanID, sampling flag, and other information via request headers to Service B, so that the Span created by Service B can be attached to the same Trace tree.

OpenTelemetry defines two standard propagation formats:

W3C TraceContext (recommended): Propagated via traceparent and tracestate headers, format: traceparent: 00-{trace-id}-{span-id}-{flags}
Baggage: Propagates business key-value pairs via the baggage header (e.g., user.id=12345), sharing business context across services

# W3C traceparent format example
traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01
              ↑  ↑______________________↑ ↑________________↑ ↑
            version      trace-id            parent-id       flags

For the W3C TraceContext specification, see W3C Recommendation

OpenTelemetry SDK Integration

OpenTelemetry (OTel) is the CNCF’s observability standard, unifying the API and SDK for three signals: Traces, Metrics, and Logs. Jaeger natively supports receiving data via the OTLP protocol.

Go Service Integration

Install dependencies:

go get go.opentelemetry.io/otel \
       go.opentelemetry.io/otel/sdk \
       go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc \
       go.opentelemetry.io/otel/trace \
       go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp \
       go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc

Initialize the Tracer Provider:

package tracing

import (
	"context"
	"time"

	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
	"go.opentelemetry.io/otel/propagation"
	"go.opentelemetry.io/otel/sdk/resource"
	sdktrace "go.opentelemetry.io/otel/sdk/trace"
	semconv "go.opentelemetry.io/otel/semconv/v1.24.0"
)

func InitTracer(ctx context.Context, serviceName, otelEndpoint string) (*sdktrace.TracerProvider, error) {
	// Create OTLP gRPC exporter, pointing to OTel Collector or Jaeger
	exporter, err := otlptracegrpc.New(ctx,
		otlptracegrpc.WithEndpoint(otelEndpoint),
		otlptracegrpc.WithInsecure(),
	)
	if err != nil {
		return nil, err
	}

	// Define service-level resource info, usable as filters in Jaeger UI
	res, _ := resource.New(ctx,
		resource.WithAttributes(
			semconv.ServiceName(serviceName),
			semconv.ServiceVersion("1.0.0"),
			semconv.DeploymentEnvironment("production"),
		),
	)

	tp := sdktrace.NewTracerProvider(
		sdktrace.WithBatcher(exporter,
			sdktrace.WithBatchTimeout(5*time.Second),
			sdktrace.WithMaxExportBatchSize(512),
		),
		sdktrace.WithResource(res),
		// Set sampler (can also use Remote/Adaptive Sampling)
		sdktrace.WithSampler(sdktrace.TraceIDRatioBased(0.1)),
	)

	otel.SetTracerProvider(tp)
	// Use W3C TraceContext as the standard propagation format
	otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(
		propagation.TraceContext{},
		propagation.Baggage{},
	))

	return tp, nil
}

HTTP service auto-instrumentation — via the otelhttp middleware, no business code changes needed:

package main

import (
	"context"
	"log"
	"net/http"

	"go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
)

func main() {
	ctx := context.Background()
	tp, err := tracing.InitTracer(ctx, "order-service", "jaeger:4317")
	if err != nil {
		log.Fatal(err)
	}
	defer tp.Shutdown(ctx)

	mux := http.NewServeMux()
	mux.HandleFunc("/api/orders", handleGetOrders)

	// Wrap with otelhttp.NewHandler to automatically create a Span for each request
	// and extract trace context from inbound request headers
	wrappedHandler := otelhttp.NewHandler(mux, "order-service")

	log.Println("listening on :8080")
	http.ListenAndServe(":8080", wrappedHandler)
}

func handleGetOrders(w http.ResponseWriter, r *http.Request) {
	// Get the current span from context and add business tags
	span := otelhttp.SpanFromContext(r.Context())
	span.SetAttributes(attribute.String("user.id", r.URL.Query().Get("uid")))

	// When calling downstream services, otelhttp automatically injects the traceparent header
	resp, err := otelhttp.Get(r.Context(), "http://user-service:8081/api/user")
	// ...
}

Python Service Integration

# pip install opentelemetry-distro opentelemetry-exporter-otlp
# opentelemetry-bootstrap -a install

from opentelemetry import trace
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.flask import FlaskInstrumentor

# Initialize Tracer
resource = Resource.create({
    "service.name": "user-service",
    "service.version": "1.0.0",
    "deployment.environment": "production",
})

provider = TracerProvider(resource=resource)
provider.add_span_processor(
    BatchSpanProcessor(
        OTLPSpanExporter(endpoint="jaeger:4317", insecure=True),
        max_export_batch_size=512,
    )
)
trace.set_tracer_provider(provider)

# Flask auto-instrumentation: each HTTP request automatically generates a Server Span
app = Flask(__name__)
FlaskInstrumentor().instrument_app(app)

@app.route("/api/user")
def get_user():
    # Manually create a child Span for DB query
    tracer = trace.get_tracer(__name__)
    with tracer.start_as_current_span("mysql.query.user") as span:
        span.set_attribute("db.system", "mysql")
        span.set_attribute("db.statement", "SELECT * FROM users WHERE id=?")
        user = db.query("SELECT * FROM users WHERE id=?", uid)
        return jsonify(user)

gRPC Context Propagation

Context propagation for gRPC calls is implemented via interceptors, ensuring traceparent is automatically injected and extracted in metadata:

// gRPC Server: register otelgrpc interceptor
import "go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc"

server := grpc.NewServer(
    grpc.StatsHandler(otelgrpc.NewServerHandler()),
)

// gRPC Client: also auto-injects context via StatsHandler
conn, _ := grpc.Dial("user-service:50051",
    grpc.WithStatsHandler(otelgrpc.NewClientHandler()),
)

This way, gRPC requests from Service A to Service B automatically carry traceparent metadata, and Service B’s Span is automatically attached under Service A’s Span.

Jaeger Deployment

Docker Compose Quick Deployment

The simplest deployment uses the All-in-One image (suitable for dev/testing), with built-in Collector + Query + in-memory storage:

# docker-compose.yml
version: '3.8'

services:
  jaeger:
    image: jaegertracing/all-in-one:1.60
    ports:
      - "16686:16686"   # Jaeger UI
      - "4317:4317"     # OTLP gRPC
      - "4318:4318"     # OTLP HTTP
      - "5778:5778"     # Sampling configuration
    environment:
      - COLLECTOR_OTLP_ENABLED=true
      - LOG_LEVEL=info
    restart: unless-stopped

After starting, visit http://localhost:16686 to access the Jaeger UI.

Kubernetes Production Deployment

For production, the Jaeger Operator is recommended, supporting automatic Sidecar injection and flexible storage backend configuration:

# jaeger-operator.yaml
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
  name: jaeger-prod
  namespace: observability
spec:
  strategy: production
  storage:
    type: elasticsearch
    options:
      es:
        server-urls: https://elasticsearch:9200
        index-prefix: jaeger
    secretName: jaeger-es-secret
  # Enable OTLP receiving port
  collector:
    options:
      collector.otlp.enabled: true
  query:
    options:
      query.base-path: /jaeger
  ingress:
    enabled: true
    hosts:
      - jaeger.sre.wang
  # Sampling configuration
  sampling:
    options:
      sampling.type: remote
      sampling.strategies-file: /etc/jaeger/sampling.json

Deploy the Operator and instance:

# Install Jaeger Operator
kubectl create namespace observability
kubectl apply -f https://github.com/jaegertracing/jaeger-operator/releases/download/v1.60.0/operator.yaml -n observability

# Deploy Jaeger instance
kubectl apply -f jaeger-operator.yaml

Sampling Strategies

In high-traffic systems, tracing every request generates enormous data volume and performance overhead. Sampling strategies determine which Traces are recorded and reported.

Const Sampling

100% sampling or no sampling at all, only suitable for development and debugging:

sdktrace.WithSampler(sdktrace.AlwaysSample())   // Full
sdktrace.WithSampler(sdktrace.NeverSample())     // Disabled

Probabilistic Sampling

Ratio-based sampling, e.g., 10% of requests are traced:

sdktrace.WithSampler(sdktrace.TraceIDRatioBased(0.1))

Remote Sampling

Remote Sampling allows the Collector to dynamically push sampling strategies without restarting services. This is the recommended approach for production.

First, configure the strategy file on the Jaeger Collector side:

// /etc/jaeger/sampling.json
{
  "service_strategies": [
    {
      "service": "order-service",
      "operation_strategies": [
        {
          "operation": "GET /api/orders",
          "probabilistic": { "samplingRate": 0.5 }
        },
        {
          "operation": "POST /api/orders",
          "probabilistic": { "samplingRate": 1.0 }
        }
      ],
      "default_strategy": {
        "probabilistic": { "samplingRate": 0.1 }
      }
    },
    {
      "service": "user-service",
      "default_strategy": {
        "probabilistic": { "samplingRate": 0.05 }
      }
    }
  ],
  "default_strategy": {
    "probabilistic": { "samplingRate": 0.01 }
  }
}

Using Remote Sampling on the Go side requires importing jaegerclient or implementing it via the OTel Collector’s sampling extension. At the OTel SDK level, it’s recommended to report via OTLP and let the Collector perform tail-based sampling.

Adaptive Sampling

Jaeger’s unique Adaptive Sampling automatically adjusts sampling rates based on the QPS of each operation: high-QPS operations get lower sampling rates, low-QPS operations get higher rates, ensuring every operation type has sufficient samples for analysis.

# Enable Adaptive Sampling on Jaeger Collector
SAMPLING_TYPE=adaptive
SAMPLING_TARGET_SAMPLES_PER_SECOND=1.0

For Adaptive Sampling principles and usage, see Jaeger Sampling Documentation

Practical: Microservice Chain Analysis and Bottleneck Identification

Scenario Description

An e-commerce order request involves 4 services:

Gateway → OrderService → UserService (gRPC)
                    → InventoryService (HTTP)
                    → PaymentService (gRPC)

Users report intermittent timeouts on the order endpoint (P99 reaching 5s), but no errors appear in individual service logs.

Analysis Process

Filter Traces in Jaeger UI: Filter by Service=gateway, Operation=POST /api/orders, min duration=3s to find slow request Traces
View the Trace tree: Expand the Trace details and find the longest-running Span
Identify the bottleneck:

Gateway: POST /api/orders           5230ms
├── OrderService: CreateOrder       5200ms ← Most of the time is here
│   ├── UserService: GetUser          15ms  ✓ Normal
│   ├── InventoryService: Lock        20ms  ✓ Normal
│   └── PaymentService: Charge      5100ms ← Bottleneck!
│       └── DB: UPDATE transactions 5080ms ← Root cause: DB row lock wait

Confirm root cause: The PaymentService Span Tag shows db.statement=UPDATE transactions WHERE... and db.duration=5080ms. Further investigation of PaymentService’s DB monitoring reveals a surge in pg_locks with locktype=transactionid during that time period, confirming concurrent orders causing lock waits on the same row.

Span Tags Best Practices

To facilitate troubleshooting, add semantic tags at critical paths:

// HTTP calls
span.SetAttributes(
    semconv.HTTPMethod("POST"),
    semconv.HTTPRoute("/api/orders"),
    semconv.HTTPStatusCode(200),
    attribute.Int("order.amount", 299),
)

// DB queries
span.SetAttributes(
    semconv.DBSystem("postgresql"),
    semconv.DBStatement("UPDATE transactions SET status=? WHERE id=?"),
    attribute.Int("db.rows_affected", 1),
)

// Error recording
import "go.opentelemetry.io/otel/codes"

span.SetStatus(codes.Error, "db connection timeout")
span.RecordError(err)
span.SetAttributes(attribute.String("error.type", "ConnectionTimeout"))

Trace-Based Alerting

Combined with Prometheus + Tempo/Loki, Trace metrics can be turned into alerts. The OTel Collector provides a spanmetrics connector that automatically converts Spans into Prometheus metrics:

# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

connectors:
  spanmetrics:
    histogram:
      explicit:
        buckets: [10ms, 50ms, 100ms, 500ms, 1s, 5s, 10s]

exporters:
  prometheus:
    endpoint: 0.0.0.0:8889
  otlp/jaeger:
    endpoint: jaeger:4317
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [spanmetrics, otlp/jaeger]
    metrics/spanmetrics:
      receivers: [spanmetrics]
      exporters: [prometheus]

This allows querying traces_span_metrics_duration_seconds_bucket in Prometheus and configuring alerts:

# P99 latency alert
histogram_quantile(0.99, sum(rate(
  traces_span_metrics_duration_seconds_bucket{
    service_name="order-service",
    span_name="POST /api/orders"
  }[5m]
)) by (le)) > 3

Summary

Distributed tracing is the key pillar among the three pillars of observability (Metrics, Logs, Traces) that connects the other two. Practice recommendations:

Unified propagation format: Use W3C TraceContext across the entire chain to avoid context breaks between heterogeneous systems
Standardized semantics: Follow OpenTelemetry Semantic Conventions for naming Span Tags to ensure queryability
Tiered sampling: High sampling (or full sampling) for critical paths, low sampling for non-critical; use Remote/Adaptive Sampling for dynamic adjustment
Correlate Metrics and Logs: Inject TraceID into logs (e.g., log.WithField("trace_id", span.SpanContext().TraceID())) to enable bidirectional log-trace navigation
Security awareness: Do not put sensitive information (passwords, tokens, PII) in Span Tags and Baggage — this data is persisted to the storage backend

OpenTelemetry official documentation: https://opentelemetry.io/docs/

References & Acknowledgments

This article referenced the following materials during writing. We thank the original authors for their contributions:

Official Documentation — Jaeger Project, referenced for Official Documentation
W3C Recommendation — W3, referenced for W3C Recommendation
Jaeger Sampling Documentation — Jaeger Project, referenced for Jaeger Sampling Documentation
opentelemetry.io — OpenTelemetry Authors, referenced for docs

Why Distributed Tracing#

Core Concepts of Distributed Tracing#

Trace#

Span#

Context Propagation#

OpenTelemetry SDK Integration#

Go Service Integration#

Python Service Integration#

gRPC Context Propagation#

Jaeger Deployment#

Docker Compose Quick Deployment#

Kubernetes Production Deployment#

Sampling Strategies#

Const Sampling#

Probabilistic Sampling#

Remote Sampling#

Adaptive Sampling#

Practical: Microservice Chain Analysis and Bottleneck Identification#

Scenario Description#

Analysis Process#

Span Tags Best Practices#

Trace-Based Alerting#

Summary#

References & Acknowledgments#