Debugging OpenTelemetry in AWS Lambda: Fixing Broken Traces Across Worker Threads

Back in July, we encountered a tricky observability bug while running an asynchronous compute pipeline inside AWS Lambda processing data from Amazon S3.

We had instrumented our workload with OpenTelemetry (OTel) to gain granular insights into execution bottlenecks, network latency, and compute time. But when we opened our tracing dashboard, instead of a clean, cohesive flamegraph showing our execution flow, we were greeted with total chaos: span explosion, orphaned traces, and completely disconnected parent-child spans.

Here is what went wrong, why our concurrency model broke OpenTelemetry’s default behavior, and how we solved it using distributed tracing context propagation.

1. Context & Architecture

Our serverless pipeline processes large objects retrieved from S3:

1. Lambda Invocation: An S3 object notification triggers the Lambda handler on the main runtime thread.
2. Concurrent Compute: Because each S3 payload contains hundreds of independent tasks, the handler dispatches these compute tasks across a pool of background worker threads to leverage multi-core Lambda compute efficiently.
3. Task Completion & Sinks: Each worker performs its computation, writes results, and signals completion.

1
2
3
4
5

[S3 Event] ──> [Lambda Handler (Main Thread)]
                     │
                     ├── Dispatch ──> [Worker Thread 1: Process Task A]
                     ├── Dispatch ──> [Worker Thread 2: Process Task B]
                     └── Dispatch ──> [Worker Thread 3: Process Task C]

To monitor performance, the Lambda handler initiated a root trace (lambda-s3-compute), and each worker task was supposed to create a child span (process-task) linked directly to that parent.

2. The Symptoms: The “Broken Trace” Mystery

When inspecting traces in our telemetry backend (Jaeger / Honeycomb), two bizarre problems showed up:

Issue A: Disconnected Orphan Traces

Instead of seeing one single trace ID containing the root handler span and multiple nested child spans, we saw dozens of separate root traces:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

WHAT WE EXPECTED:
└── [Trace ID: 4bf92f3577b34da6a3ce929d0e0e4736]
    └── lambda-s3-compute (Parent Span)
        ├── process-task (Worker 1)
        ├── process-task (Worker 2)
        └── process-task (Worker 3)

WHAT ACTUALLY HAPPENED:
├── [Trace ID: 4bf92f3577b34da6] -> lambda-s3-compute (Orphaned Parent)
├── [Trace ID: 8a1b92c431d87e01] -> process-task (Worker 1 as a new Root!)
├── [Trace ID: 5f2a1b9c84e12019] -> process-task (Worker 2 as a new Root!)
└── [Trace ID: 3e9d81a4b5c77204] -> process-task (Worker 3 as a new Root!)

Issue B: Span Explosion & Redundant Telemetry

Because child tasks were not tethered to their parent, tasks that spawned sub-tasks or HTTP requests generated runaway spans with distinct IDs. Telemetry ingestion costs spiked, and searching for the trace of a single S3 file execution was virtually impossible because none of the spans shared a common trace context.

3. The Investigation & Root Cause

Where was the context getting lost?

In the main Lambda handler, the span was active:

1
2
3
4
5
6
7
8
9

// Main handler thread
Tracer tracer = openTelemetry.getTracer("s3-compute");
Span parentSpan = tracer.spanBuilder("lambda-s3-compute").startSpan();
try (Scope scope = parentSpan.makeCurrent()) {
    // Context is active HERE on the main thread
    dispatchTasksConcurrently(tasks);
} finally {
    parentSpan.end();
}

Inside the worker threads, we were starting child spans like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

// Worker thread (inside ExecutorService / Runnable)
public void run() {
    // We expected OTel to automatically know the parent!
    Span childSpan = tracer.spanBuilder("process-task").startSpan();
    try {
        processItem();
    } finally {
        childSpan.end();
    }
}

The “Aha!” Moment: Thread Boundaries Need Context Propagation

In the Java OpenTelemetry SDK, the default context storage is thread-local. This is an implementation detail of Java context storage; other language SDKs use their own context-local mechanisms.

• When parentSpan.makeCurrent() is called, the context is made current on the main thread for the scope in which it is active.
• When tasks are submitted to a manually managed ExecutorService, the worker thread does not automatically inherit the parent thread’s context.
• When tracer.spanBuilder("process-task").startSpan() executes inside the worker thread, Context.current() does not contain the parent span. The span builder therefore creates a new root span:

  “There is no active span on this thread. I must create a brand-new Trace ID and treat this as a brand-new Root Span!”

This explained why the worker spans were severed from the parent and appeared under unrelated trace IDs.

4. The Fix: Explicit Context Propagation Across Threads

To fix this, we applied the fundamental rule of distributed tracing: When crossing an execution boundary (network or thread), you must explicitly propagate the Context.

The Solution: Wrapping Runnables with OTel Context

OpenTelemetry provides built-in utilities to capture the active context from the parent thread and attach it to the worker execution.

Before (Broken):

1
2
3
4
5

// Parent thread dispatches without context:
executorService.submit(() -> {
    // No parent span is current on this worker thread.
    doWork();
});

After (Fixed with Context.current().wrap()):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

// 1. Capture the current context on the main thread
Context parentContext = Context.current();

for (Task task : tasks) {
    // 2. Wrap the Runnable so it carries the parent context into the worker thread
    Runnable wrappedTask = parentContext.wrap(() -> {
        // 3. Inside worker: Context is now restored!
        Span childSpan = tracer.spanBuilder("process-task")
            .startSpan();
        try (Scope scope = childSpan.makeCurrent()) {
            executeTask(task);
        } finally {
            childSpan.end();
        }
    });

    executorService.submit(wrappedTask);
}

Alternative: Using OpenTelemetry Context in Custom Handlers

If you’re passing state objects between threads, you can also pass the parent Context explicitly:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

public void processInWorker(Context parentContext, Task task) {
    // Explicitly bind the parent context to the span builder
    Span childSpan = tracer.spanBuilder("process-task")
        .setParent(parentContext)
        .startSpan();

    try (Scope scope = childSpan.makeCurrent()) {
        execute(task);
    } finally {
        childSpan.end();
    }
}

5. The Outcome

Once we deployed the fix:

1. Unified Traces: Every S3 invocation produced exactly one Trace ID.
2. Proper Tree Hierarchy: All worker threads appeared as neat, concurrent parallel child bars under the main handler span in the flamegraph.
3. Accurate Bottleneck Detection: We could instantly identify which specific S3 task worker was bottlenecked on I/O vs. compute.
4. Clean Telemetry Bills: Zero runaway root spans or detached noise.

6. Key Takeaways

1. Do Not Assume Context Crosses Thread Pools: With manually managed Java executors, do not assume the current OpenTelemetry context will travel with a task unless instrumentation or explicit propagation is in place.
2. Use Context.current().wrap(): OpenTelemetry’s Context.wrap() is one straightforward way to capture and restore context around a Runnable or Callable. Instrumented executors can also perform this propagation for you.
3. Always End Spans in finally Blocks: Ensure span.end() is guaranteed to execute, even on unhandled exceptions, to prevent memory leaks and dangling spans.
4. Inspect Flamegraphs Early: Don’t just test if telemetry is arriving; verify that the Trace ID and Parent Span ID relationships are mathematically connected.