APM Tool Selection: A Practical Guide from Open-Source to Commercial Solutions

Overview

Have you ever encountered this situation: users report “the system is slow,” you open Grafana and check a bunch of dashboards—CPU is fine, memory is fine, network is fine—but users insist it’s slow. What you need at this point is not more metric dashboards, but a complete request trace—from the moment the user clicks a button to when the database returns results, showing exactly how long each hop took and where it got stuck.

That’s what APM (Application Performance Monitoring) does.

Simply put: logs tell you what happened, metrics tell you whether the system is healthy, and APM tells you why it’s slow, where it’s slow, and who it affects. Each serves a different purpose, and you need all three.

This article takes a practical selection-oriented approach, breaking down the architecture differences, applicable scenarios, and pitfalls of mainstream open-source and commercial APM tools. No hype, no bashing—every tool has its sweet spot. The key is matching it to your team size, tech stack, and budget.

What Problem Does APM Solve

Let’s first clarify why you need APM before jumping into tool comparisons.

The “Black Box” of Microservice Call Chains

In the monolithic era, a request stayed within a single process from entry to database. You could set a breakpoint and debug. After microservice decomposition, a single user request might traverse: API Gateway → Auth Service → Order Service → Payment Service → Message Queue → Inventory Service → Database, with Redis cache and third-party API calls sprinkled in between.

Any link in this chain slows down, and the whole thing feels slow. But logs only show you a single service’s perspective—you can’t string the full chain together. It’s like going to a hospital where each department says you’re fine, but you still feel sick. APM is the “general practitioner” who puts all your test results together.

Three Core Capabilities of APM

Distributed tracing is APM’s most essential capability. It works by generating a unique Trace ID at the request entry point, then propagating it through HTTP headers or RPC context to downstream services. Each service records a Span (think of it as a node in the chain) during its processing, and together they form a complete call tree.

Key Concepts at a Glance

Sampling strategy is critical. At high traffic volumes, full tracing will exhaust your storage and CPU. The common approach is head-based sampling—decide at the request entry whether to record, then either trace the entire chain or skip it entirely. Tail-based sampling is more granular—decide at chain completion based on conditions (e.g., latency exceeded threshold, errors occurred), but it’s more complex to implement and requires an intermediate layer to buffer complete chains.

Open-Source APM Landscape

The open-source APM landscape in 2026 has settled into a fairly clear pattern. By feature coverage, tools fall into three categories:

1. All-in-one APM: Tracing + metrics + alerting in one package, represented by SkyWalking, Pinpoint
2. Tracing specialists: Only handle Trace, need Prometheus + Grafana for metrics, represented by Jaeger, Zipkin
3. Composable observability stacks: Prometheus + Grafana + Loki + Tempo (the LGTM stack)—flexible but higher integration cost

Five Open-Source Solutions at a Glance

Apache SkyWalking

SkyWalking is an Apache Foundation top-level project with high adoption in China. Its core selling point is out-of-the-box functionality—install the OAP (Observability Analysis Platform) server + Agent, and you get service topology maps, distributed tracing, metric monitoring, and alerting automatically, without needing to set up Prometheus separately.

Architecturally, SkyWalking uses Agents for bytecode enhancement on the application side (JavaAgent for Java, gRPC manual instrumentation for other languages), sends data via gRPC/HTTP to the OAP server, which handles aggregation, analysis, and storage. Storage options include Elasticsearch, BanyanDB (SkyWalking’s purpose-built time-series database), and H2 (testing only).

Strengths:

• Auto-discovered topology maps—service dependencies at a glance
• One-stop solution, no need to assemble components
• Active domestic community, Chinese-friendly documentation
• Service Mesh support (Istio/Envoy data plane)

Pitfalls:

• OAP server is memory-hungry—production needs at least 8GB
• ES storage degrades under high cardinality; BanyanDB is still maturing
• Non-Java agents are significantly weaker than the Java agent
• Configuration leans heavily on XML/YAML, with a steep learning curve

A typical SkyWalking deployment:

# docker-compose-skywalking.yml
version: '3.8'

services:
  oap:
    image: apache/skywalking-oap-server:10.1.0
    ports:
      - "11800:11800"  # gRPC receives Agent data
      - "12800:12800"  # HTTP REST API
    environment:
      SW_STORAGE: elasticsearch
      SW_STORAGE_ES_CLUSTER_NODES: elasticsearch:9200
      SW_CORE_RECORD_DATA_TTL: 7        # Trace data retention: 7 days
      SW_CORE_METRICS_DATA_TTL: 30      # Metric data retention: 30 days
    depends_on:
      - elasticsearch
    restart: unless-stopped

  ui:
    image: apache/skywalking-ui:10.1.0
    ports:
      - "8080:8080"
    environment:
      SW_OAP_ADDRESS: http://oap:12800
    depends_on:
      - oap
    restart: unless-stopped

  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.13.0
    environment:
      - discovery.type=single-node
      - xpack.security.enabled=false
      - "ES_JAVA_OPTS=-Xms2g -Xmx2g"
    volumes:
      - es-data:/usr/share/elasticsearch/data
    restart: unless-stopped

volumes:
  es-data:

Java application integration requires just one parameter:

# Mount SkyWalking Agent when starting a Java application
java -javaagent:/path/to/skywalking-agent.jar \
     -Dskywalking.agent.service_name=order-service \
     -Dskywalking.collector.backend_service=oap:11800 \
     -jar order-service.jar

Go applications require manual instrumentation (SkyWalking Go Agent is still developing), using OTel SDK + SkyWalking exporter:

package main

import (
    "context"
    "log"
    
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/trace"
)

// Initialize tracer, sending data to SkyWalking OAP
func initTracer() func() {
    // In production, configure OTLP exporter pointing to SkyWalking OAP's OTLP receiving port
    // SkyWalking 9.x+ natively supports OTLP protocol
    tp, err := initOTLPProvider("oap:11800", "order-service")
    if err != nil {
        log.Fatal(err)
    }
    otel.SetTracerProvider(tp)
    return func() { tp.Shutdown(context.Background()) }
}

// Create span in HTTP handler
func handleOrder(ctx context.Context, orderID string) error {
    ctx, span := tracer.Start(ctx, "handleOrder")
    defer span.End()
    
    span.SetAttributes(attribute.String("order.id", orderID))
    
    // Call downstream payment service, trace context auto-propagates
    if err := callPaymentService(ctx, orderID); err != nil {
        span.RecordError(err)
        return err
    }
    return nil
}

Jaeger

Jaeger (German for “hunter”) is a distributed tracing project open-sourced by Uber and later donated to CNCF. It does one thing—Trace storage, querying, and visualization. No metrics, no alerting.

Jaeger’s positioning is clear: if you already have Prometheus + Grafana for metrics and just need a tracing backend, Jaeger is the cleanest choice. It doesn’t try to do everything, but what it does, it does well.

Strengths:

• CNCF graduated project, integrates well with Kubernetes ecosystem
• Native OpenTelemetry protocol (OTLP) support
• Clean UI, highly readable trace waterfall diagrams
• Adaptive Sampling—automatically adjusts sampling rate based on traffic

Pitfalls:

• Only handles Trace; metrics and logs need separate solutions
• Storage backend selection is a headache—ES is heavy, Cassandra is complex to operate, Badger is single-machine only
• Community activity is lower than SkyWalking

Jaeger supports direct OTLP ingestion—use OpenTelemetry SDK for instrumentation and send directly to Jaeger:

# Jaeger all-in-one deployment (testing only)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: jaeger
spec:
  replicas: 1
  selector:
    matchLabels:
      app: jaeger
  template:
    metadata:
      labels:
        app: jaeger
    spec:
      containers:
      - name: jaeger
        image: jaegertracing/all-in-one:1.60
        ports:
        - containerPort: 16686  # UI
        - containerPort: 4317    # OTLP gRPC
        - containerPort: 4318    # OTLP HTTP
        env:
        - name: COLLECTOR_OTLP_ENABLED
          value: "true"
        - name: SPAN_STORAGE_TYPE
          value: badger
        - name: BADGER_EPHEMERAL
          value: "false"
        - name: BADGER_DIRECTORY_VALUE
          value: /data/values
        - name: BADGER_DIRECTORY_KEY
          value: /data/keys
        volumeMounts:
        - name: data
          mountPath: /data
      volumes:
      - name: data
        persistentVolumeClaim:
          claimName: jaeger-pvc

Zipkin

Zipkin is a distributed tracing system open-sourced by Twitter, predating Jaeger. Its design philosophy is “good enough”—not feature-rich but stable, simple to deploy.

Honestly, I don’t recommend Zipkin for new projects. Jaeger fully covers Zipkin’s capabilities, and Jaeger has native OTLP support with a more active ecosystem. Zipkin’s advantage is its history and broad SDK ecosystem, but new projects are better off with Jaeger or Tempo.

Grafana Tempo

Tempo is Grafana Labs’ high-performance Trace backend, with one key selling point: object storage instead of a database.

Traditional Trace storage uses ES or Cassandra—expensive and operationally heavy. Tempo stores Trace data on S3/GCS/MinIO object storage, cutting costs by an order of magnitude with virtually unlimited capacity. Queries retrieve by Trace ID directly, without full-text search (this is the core difference from Jaeger).

Strengths:

• Extremely low storage cost—a few dollars per month on S3 for massive Trace volumes
• Deep Grafana integration—trace, metrics, and logs in one view
• Simple architecture—only ingester + querier + compactor

Pitfalls:

• Must know the Trace ID to query—no searching by service name + time range for Trace lists (improved significantly with TraceQL since v2.0)
• Depends on object storage; local deployment requires MinIO

Tempo is ideal for teams already using the Grafana suite. If your metrics are on Prometheus and logs on Loki, Tempo is the natural choice for tracing:

# Tempo + MinIO deployment
server:
  http_listen_port: 3200

distributor:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 0.0.0.0:4317
        http:
          endpoint: 0.0.0.0:4318

ingester:
  max_block_duration: 5m

compactor:
  compaction:
    block_retention: 48h

storage:
  trace:
    backend: s3
    s3:
      bucket: tempo-traces
      endpoint: minio:9000
      access_key: minioadmin
      secret_key: minioadmin
      insecure: true

Pinpoint

Pinpoint is an APM open-sourced by Korea’s Naver, characterized by pure Java, zero intrusion. It uses bytecode enhancement for automatic instrumentation—Java applications just attach the Agent, no code changes needed.

If you’re a pure Java shop, Pinpoint’s zero-intrusion experience is genuinely good. But the moment you have Go, Python, or Node.js services, Pinpoint can’t help. Its storage dependency on HBase also adds operational complexity. For new projects, I’d recommend SkyWalking instead—same bytecode enhancement approach, but better multi-language support and community activity.

Commercial APM Comparison

The core advantage of commercial APM is peace of mind—no need to operate storage backends, professional teams handle anomaly detection algorithms, and integration is tighter. But the price tag isn’t cheap either.

Datadog’s Q3 2025 quarterly revenue was $885.7M, with a market cap of approximately $40.2B, serving 95% of Fortune 500 companies globally. It has been named a Gartner Magic Quadrant Leader for Observability Platforms for five consecutive years. Source: 2026 Observability Vendor Selection Guide

Datadog

Datadog is currently the hottest commercial observability platform, bar none. Its killer feature is integration breadth—850+ out-of-the-box integrations covering AWS/GCP/Azure all major cloud services, Kubernetes, databases, message queues, APM, logs, and security.

Datadog’s APM uses the dd-trace library for auto-instrumentation, supporting Java/Go/Python/Node.js/.NET/PHP/Ruby. Data goes to the Datadog SaaS backend, and the UI lets you drill down from metrics to traces to logs seamlessly.

But Datadog’s pricing model warrants caution: per-host + per-module billing. For 50 hosts with APM + logs + infrastructure monitoring, monthly costs easily reach $3000-5000. Also, all data is stored on Datadog’s SaaS, so domestic enterprises need to consider cross-border data compliance.

Dynatrace

Dynatrace’s core differentiator is the Davis AI engine—it’s not simple threshold alerting, but causal analysis for automatic root cause identification. For example, if a service slows down, Davis can tell you “because a dependent database query slowed down, and the query slowed down because an index was deleted.”

OneAgent is Dynatrace’s data collection method—a single agent automatically covers the full stack from infrastructure to application code to user experience. Zero-configuration auto-discovery after installation, which is very appealing for large enterprises.

However, Dynatrace is highly proprietary—OneAgent is closed-source, data formats aren’t open. Once you’re in, migration costs are extremely high. At approximately $69/host/month, large-scale deployments get expensive.

New Relic

New Relic is a veteran in the APM space with excellent developer experience. NRQL (New Relic Query Language) is a SQL-like query language with high flexibility. New Relic was also among the first commercial vendors to embrace OpenTelemetry—their OTel-native architecture provides better data portability.

However, New Relic’s infrastructure monitoring and database deep monitoring are relatively weak. MySQL slow query analysis, Redis cache hit rate—these DBA-centric scenarios aren’t as comprehensive as Datadog. Per-user billing at scale can also spiral out of control.

OpenTelemetry: The Vendor-Neutral Future

You can’t discuss APM selection without addressing OpenTelemetry (OTel). It’s CNCF’s second most active project (after Kubernetes), aiming to unify the data collection standards for all three observability pillars (Trace/Metrics/Logs).

Why OTel Matters

Before OTel, every APM vendor had their own SDK: Jaeger used Jaeger client, Zipkin used Brave/Zipkin client, SkyWalking had its own Agent. Choosing a vendor meant using their SDK, and switching vendors required rewriting all services’ instrumentation code—classic vendor lock-in.

OTel solves the instrumentation standardization problem: applications use only the OTel SDK for instrumentation, data goes out via the OTLP protocol, and the backend can be Jaeger, SkyWalking, Tempo, or Datadog—any of them. Switching backends doesn’t require application code changes.

OpenTelemetry was formed in 2019 by merging OpenTracing and OpenCensus, inheriting OpenTracing’s vendor-neutral philosophy and OpenCensus’s multi-signal capabilities. Traces Spec reached Stable in 2021, Metrics Spec in late 2021, and Logs Spec in mid-2023. Source: OpenTelemetry in Practice: Unified Standards for Cloud-Native Observability’s Three Pillars

OTel Architecture: Three-Layer Separation

┌─────────────────────────────────────────────────────────┐
│                    Application Layer                     │
│   OTel SDK auto/manual instrumentation → Span/Metric/Log│
└────────────────────────┬────────────────────────────────┘
                         │ OTLP protocol (gRPC :4317 / HTTP :4318)
                         ▼
┌─────────────────────────────────────────────────────────┐
│                  Collector Layer                         │
│   Receive → Process (filter/sample/batch) → Export      │
└─────────┬───────────────────────┬───────────────────────┘
          │                       │
          ▼                       ▼
┌──────────────────┐   ┌──────────────────────────────────┐
│  Jaeger / Tempo   │   │  Prometheus / SkyWalking / ES   │
│  (Trace backend)  │   │  (Metrics / Log backend)         │
└──────────────────┘   └──────────────────────────────────┘

OTel Collector Deployment

The OTel Collector is the core component in the OTel architecture—it’s a data relay responsible for receiving, processing, and exporting telemetry data. Production deployments should strongly consider deploying a Collector rather than having applications connect directly to backends:

# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  # Batching to reduce export request count
  batch:
    timeout: 5s
    send_batch_size: 1024
  
  # Memory limiter to prevent OOM from traffic spikes
  memory_limiter:
    check_interval: 1s
    limit_mib: 512
  
  # Tail sampling: only keep slow requests and errors
  tail_sampling:
    decision_wait: 10s
    policies:
      - name: errors
        type: status_code
        status_code:
          status_codes: [ERROR]
      - name: slow
        type: latency
        latency:
          threshold_ms: 500
      - name: random_keep
        type: probabilistic
        probabilistic:
          sampling_percentage: 10

exporters:
  # Send to Jaeger
  otlp/jaeger:
    endpoint: jaeger:4317
    tls:
      insecure: true
  
  # Send to Prometheus (metrics)
  prometheus:
    endpoint: 0.0.0.0:8889
  
  # Send to Loki (logs)
  loki:
    endpoint: http://loki:3100/loki/api/v1/push

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, tail_sampling, batch]
      exporters: [otlp/jaeger]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [prometheus]
    logs:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [loki]

Application-Side Instrumentation Example (Go)

package main

import (
    "context"
    "log"
    "net/http"
    
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/attribute"
    "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
    "go.opentelemetry.io/otel/propagation"
    "go.opentelemetry.io/otel/sdk/resource"
    sdktrace "go.opentelemetry.io/otel/sdk/trace"
    semconv "go.opentelemetry.io/otel/semconv/v1.24.0"
    "go.opentelemetry.io/otel/trace"
    "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
)

func initTracer(ctx context.Context, serviceName string) func() {
    // Create OTLP gRPC exporter, pointing to Collector
    exporter, err := otlptracegrpc.New(ctx,
        otlptracegrpc.WithEndpoint("otel-collector:4317"),
        otlptracegrpc.WithInsecure(),
    )
    if err != nil {
        log.Fatalf("failed to create exporter: %v", err)
    }
    
    // Configure resource info (service name, instance ID, etc.)
    res, _ := resource.New(ctx,
        resource.WithAttributes(
            semconv.ServiceName(serviceName),
            semconv.ServiceVersion("v1.2.0"),
            semconv.DeploymentEnvironment("production"),
        ),
    )
    
    // Create TracerProvider
    tp := sdktrace.NewTracerProvider(
        sdktrace.WithBatcher(exporter),
        sdktrace.WithResource(res),
        // Head-based sampling: 10% sampling rate
        sdktrace.WithSampler(sdktrace.TraceIDRatioBased(0.1)),
    )
    otel.SetTracerProvider(tp)
    otel.SetTextMapPropagator(propagation.TraceContext{})
    
    return func() {
        _ = tp.Shutdown(ctx)
    }
}

func main() {
    ctx := context.Background()
    shutdown := initTracer(ctx, "api-gateway")
    defer shutdown()
    
    tracer := otel.Tracer("api-gateway")
    
    // Use otelhttp for automatic HTTP server instrumentation
    handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        ctx := r.Context()
        span := trace.SpanFromContext(ctx)
        span.SetAttributes(attribute.String("user.agent", r.UserAgent()))
        
        // Call downstream service, TraceContext auto-propagates
        callDownstream(ctx, "http://order-service:8080/api/orders")
        w.Write([]byte("OK"))
    })
    
    wrapped := otelhttp.NewHandler(handler, "api-gateway")
    http.ListenAndServe(":8080", wrapped)
}

func callDownstream(ctx context.Context, url string) {
    // otelhttp.NewClient automatically injects Trace Header
    client := http.Client{
        Transport: otelhttp.NewTransport(http.DefaultTransport),
    }
    req, _ := http.NewRequestWithContext(ctx, "GET", url, nil)
    resp, err := client.Do(req)
    if err != nil {
        log.Printf("downstream call failed: %v", err)
        return
    }
    defer resp.Body.Close()
}

Selection Decision Framework

With so many tools covered, how do you actually choose? Here’s a six-dimension selection framework, ordered by priority:

Dimension 1: Tech Stack Match

Dimension 2: Operational Complexity

The biggest cost of open-source APM isn’t the license—it’s operations. You have to manage storage backends, ensure high availability, and plan capacity:

Dimension 3: Storage Cost

Trace data volume is substantial. A medium-scale microservice cluster (50 services, 100M requests/day) at 10% sampling generates 50-200GB of Trace data per day.

Dimension 4: Compliance and Data Sovereignty

Data compliance requirements for domestic enterprises are getting stricter. If you’re in finance, government, or healthcare, cross-border data is a red line:

Dimension 5: Team Size and Capability

Dimension 6: TCO (Total Cost of Ownership)

Don’t just look at license fees. Open-source TCO includes:

• Storage costs (ES cluster / S3 storage)
• Operations headcount (at least 0.5 FTE dedicated)
• Hardware costs (OAP/Collector nodes)
• Training costs (team learning curve)

For a 50-host cluster, open-source annual TCO is approximately 150,000-300,000 RMB (including personnel), while commercial solutions run 300,000-600,000 RMB. The cost advantage of open-source only becomes apparent at larger scale.

Production Environment Recommendations

Recommendation 1: Don’t Slack on Sampling Strategy

Full sampling sounds great in theory, but in production it’ll blow up your storage in two days. Based on my experience, a reasonable sampling strategy is:

# Recommended production sampling configuration
sampling:
  # Normal requests: head-based sampling 5-10%
  head_based:
    ratio: 0.05
  
  # Slow requests (> 500ms): keep all
  # Error requests: keep all
  # Implemented via OTel Collector tail sampling
  tail_based:
    policies:
      - type: status_code
        status_codes: [ERROR]
      - type: latency
        threshold_ms: 500
      - type: probabilistic
        sampling_percentage: 5

This controls data volume while not missing critical issues.

Recommendation 2: Collector Must Be Highly Available

The OTel Collector is the bottleneck of your data pipeline. If it goes down, all applications’ traces stop flowing. Production deployments need at least 3 Collector instances + load balancing:

# Kubernetes deployment for OTel Collector (Deployment mode)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: otel-collector
spec:
  replicas: 3
  selector:
    matchLabels:
      app: otel-collector
  template:
    metadata:
      labels:
        app: otel-collector
    spec:
      containers:
      - name: collector
        image: otel/opentelemetry-collector-contrib:0.103.0
        ports:
        - containerPort: 4317  # OTLP gRPC
        - containerPort: 4318  # OTLP HTTP
        - containerPort: 8888  # Metrics
        resources:
          requests:
            cpu: 500m
            memory: 512Mi
          limits:
            cpu: 2000m
            memory: 2Gi
        livenessProbe:
          httpGet:
            path: /health
            port: 13133
        readinessProbe:
          httpGet:
            path: /health
            port: 13133
---
apiVersion: v1
kind: Service
metadata:
  name: otel-collector
spec:
  selector:
    app: otel-collector
  ports:
  - name: otlp-grpc
    port: 4317
    targetPort: 4317
  - name: otlp-http
    port: 4318
    targetPort: 4318

Recommendation 3: Don’t Chase Perfection on Day One

I’ve seen too many teams spend two weeks agonizing over tool selection and end up with nothing deployed. The pragmatic approach:

1. Week 1: Use OTel SDK to instrument your 2-3 most critical services. Start with Jaeger all-in-one single instance for the backend.
2. Week 2: Validate trace data quality—check if chains are complete, if spans have business context.
3. Month 1: Decide backend based on actual data volume—small volume: Jaeger + Badger; large volume: Tempo + S3 or SkyWalking + ES.
4. Month 3: Integrate alerting, configure SLOs, make trace data actually serve troubleshooting.

Recommendation 4: Add Business Context to Span Tags

Purely technical traces (just HTTP method + URL + latency) are of limited value. What’s truly valuable is traces with business context:

// Good practice: include business info in Spans
func processOrder(ctx context.Context, order *Order) error {
    ctx, span := tracer.Start(ctx, "processOrder")
    defer span.End()
    
    // Key business attributes
    span.SetAttributes(
        attribute.String("order.id", order.ID),
        attribute.String("order.user_id", order.UserID),
        attribute.Float64("order.amount", order.Amount),
        attribute.String("order.status", order.Status),
    )
    
    // Record key events
    span.AddEvent("payment_initiated", trace.WithAttributes(
        attribute.String("payment.gateway", order.PaymentGateway),
    ))
    
    if err := validateOrder(ctx, order); err != nil {
        span.RecordError(err)
        span.SetStatus(codes.Error, err.Error())
        return err
    }
    
    return nil
}

This way, when searching traces in Jaeger/SkyWalking UI, you can filter directly by order.id=xxx to quickly locate the complete chain for a specific order.

Recommendation 5: Monitor Your Monitoring System

The APM system itself is a service—it needs monitoring too. At minimum, watch:

# PromQL: Monitor OTel Collector data drop rate
rate(otelcol_processor_refused_spans_total[5m]) 
/ 
(rate(otelcol_receiver_accepted_spans_total[5m]) + rate(otelcol_processor_refused_spans_total[5m]))

Recommended Solutions by Team Size

Small Team (1-5 servers)

Don’t overthink it. Use Datadog or New Relic’s free tier, or Jaeger all-in-one single instance. Your time should go to the business, not operating a monitoring system.

Mid-Size (20-100 servers)

Recommended: OTel SDK + Jaeger (or Tempo) + Prometheus + Grafana.

• Instrumentation with OTel SDK for vendor neutrality
• Trace backend: Jaeger + ES or Tempo + S3
• Metrics: continue with Prometheus
• Logs: Loki or ELK
• Unified display in Grafana

This combination’s TCO is 60-70% lower than commercial solutions, but requires 0.5-1 FTE for maintenance.

Large Scale (100+ servers)

Recommended: SkyWalking or self-built OTel + LGTM full stack.

• SkyWalking: all-in-one, fewer components to operate
• Or self-built OTel Collector + Tempo + Prometheus + Loki + Grafana for maximum customizability
• Deploy multi-region federation for cross-datacenter queries
• Consider Tail Sampling for fine-grained sampling control

Finance/Government and Other High-Compliance Scenarios

Cross-border data is prohibited, ruling out commercial SaaS. Recommended:

• SkyWalking private deployment (active domestic community, full Chinese documentation)
• Or self-built OTel + Jaeger + ES with all data stored locally
• Configure log retention per compliance requirements (typically 180+ days)

Summary

There’s no silver bullet in APM selection. I’ve seen too many teams spend big on commercial APM only to use it for topology maps; and teams who built complete observability platforms with open-source but couldn’t maintain them.

Key takeaways:

1. Adopt OTel first, choose backend later. Regardless of the final tool, standardize application-side instrumentation with OpenTelemetry SDK. This lets you switch backends without touching business code—the most important architectural decision.
2. Small teams shouldn’t self-host. For teams of 5 or fewer ops engineers, use commercial SaaS directly. The operational cost of self-hosting open-source APM far exceeds license fees.
3. SkyWalking is the default choice for domestic Java teams. Out-of-the-box, nice topology maps, good community support. But watch for OAP memory consumption and ES tuning pitfalls.
4. Tempo works for storage-cost-sensitive scenarios. S3 storage + Grafana display costs only 1/10 of ES-based solutions. But query patterns are limited—best for “query by Trace ID” use cases.
5. For commercial, look at Datadog. If data compliance isn’t a concern and budget allows, Datadog’s integration breadth and product maturity are genuinely leading. But manage costs carefully—don’t get burned by metered billing.
6. Sampling strategy determines system viability. Full sampling will overwhelm storage; reasonable sampling is key to long-term APM stability.

One final piece of experience: APM isn’t “install and forget.” It requires ongoing investment—adjusting sampling rates, refining Span tags, integrating with SLOs for alerting. An APM system that’s been running for a year is far more valuable than one just deployed, because it has accumulated enough historical baseline data. Don’t switch tools frequently—once you’ve chosen, commit and go deep.

References & Acknowledgments

The following resources were referenced during the writing of this article. We thank the original authors for their contributions:

1. 2026 Open-Source APM Selection Guide: Choosing OpenTelemetry-Native Solutions — DataBuff, open-source APM tool classification and selection dimension comparison
2. 2026 Top 5 Open-Source APM Tools Comparison Guide — Tencent Cloud Developer Community, five open-source APM tools’ features and pros/cons analysis
3. 2026 Observability Vendor Selection Guide — CSDN, commercial observability vendors (Datadog/Dynatrace/New Relic) market data and capability comparison
4. APM Tools Introduction: Agent Probes, Trace Tracking, Span Segments, Sampling — CSDN, APM core concepts and working principles
5. OpenTelemetry in Practice: Unified Standards for Cloud-Native Observability’s Three Pillars — CSDN, OpenTelemetry history and architecture design
6. APM Tool Selection Ultimate Comparison: Applications Manager vs Datadog vs New Relic — ManageEngine, commercial APM tool multi-dimensional comparison