Skip to content

fix: use DD_SERVICE for cold start spans instead of hardcoded aws.lambda - #855

Closed
joeyzhao2018 wants to merge 1 commit into
mainfrom
APMS-20524
Closed

joeyzhao2018 wants to merge 1 commit into
mainfrom
APMS-20524

Conversation

@joeyzhao2018

Copy link
Copy Markdown
Contributor

What does this PR do?

Cold start spans (aws.lambda.load, aws.lambda.import) were always created with
service="aws.lambda". They now use DD_SERVICE when set, and still fall back to
aws.lambda when it isn't.

Why this is safe

  1. Extension users already get this. Since Nov 2024 the Rust extension rewrites any
    library span whose service is aws.lambda to DD_SERVICE (datadog-lambda-extension#471,
    bottlecap/src/traces/trace_processor.rs: "Service name could be incorrectly set to
    'aws.lambda' in datadog lambda libraries"). With extension + DD_SERVICE, what reaches
    the backend doesn't change. This only fixes the Forwarder / no-extension path. Anything
    keyed on service:aws.lambda for these spans has already been broken for extension
    users for almost two years.
  2. We already did this for the function span, with no flag: Enhance Synthetic Span Service Representation #635 (Python, Jul 2025)
    and datadog-lambda-js#666 ("Remove Service Override from aws.lambda spans", Jul 2025,
    prompted by Tier-1 escalation SLES-2371). The cold start spans were missed, and no
    snapshot caught it because integration tests disable cold start tracing.
  3. Our UI expects the customer's service. The Serverless "Emitting cold start traces"
    facet / Cold Start Tracing insight looks for trace.aws.lambda.load with the
    function's resource_name + service + env (web-ui transform-lambda-entity.ts).
    With service:aws.lambda it never matches for non-extension users. Trace-view
    highlighting selects cold start spans by operation_name only, so it's unaffected.
    aws.lambda.load is still a top-level span, so trace.aws.lambda.load.* metrics keep
    being generated, now under the customer's service.
  4. Nothing internal keys on service:aws.lambda for these spans. I searched web-ui,
    dogweb, dd-go, public docs and OOTB dashboards. The docs never state the service name.
    Our cold-start regression monitoring uses aws.lambda.enhanced.init_duration, not spans.
  5. Why not gate it like fix: use DD_SERVICE for inferred spans when integration service names are removed #834? Inferred spans stand for a different entity (a queue,
    an API) whose AWS-resource service name is a real node on the service map. Cold start
    spans are the function's own in-process init work. aws.lambda isn't an entity there,
    it's a service override (dd-trace-py tags these spans _dd.base_service). SLES-2371
    was a customer escalation about exactly that kind of override.

What changes for customers

Only non-extension users with DD_SERVICE set:

  • Cold start spans move from aws.lambda to DD_SERVICE, and aws.lambda.load shows
    up as an operation on their service.
  • The phantom aws.lambda service override goes away.
  • A custom query or monitor on service:aws.lambda + operation_name:aws.lambda.load
    would stop matching. We found no evidence of such usage.

Nothing changes when DD_SERVICE is unset.

Testing

  • Unit tests for DD_SERVICE set and unset.
  • An end-to-end test with the real tracer asserting the function, load and import spans
    share the same service.
  • No snapshot changes.

🤖 Generated with Claude Code

Cold start spans (aws.lambda.load, aws.lambda.import) were always created
with service="aws.lambda". Use DD_SERVICE when set, falling back to
aws.lambda otherwise, so they land in the same service as the function span.

This matches what the Lambda extension already does by rewriting
aws.lambda spans to DD_SERVICE (datadog-lambda-extension#471), and
follows the same change made for the function span in #635.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@joeyzhao2018
joeyzhao2018 requested review from a team as code owners September 22, 2026 22:06
@datadog-datadog-prod-us1

datadog-datadog-prod-us1 Bot commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

Pipelines

❌ Errors

Your PR has failed checks. Please review the issues below and take necessary action before merging.

🚦 4 Pipeline jobs failed

DataDog/datadog-lambda-python | check-layer-size (python311-amd64)

View more details · View in GitLab

DataDog/datadog-lambda-python | check-layer-size (python312-amd64)

View more details · View in GitLab

DataDog/datadog-lambda-python | check-layer-size (python313-amd64)

View more details · View in GitLab

View all 4 failed jobs.

Useful? React with 👍 / 👎

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 47d4b26 | Docs | View more details | Give us feedback!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant