Every DVA-C02 study guide tells you two things about AWS X-Ray: it draws a service map, and annotations are indexed while metadata is not. Both are true and both are the answer to maybe one question. The other X-Ray questions in the Troubleshooting and Optimization domain (18% of the exam) are about mechanics: why a downstream Lambda function produced no trace even though the caller was traced, why a sampling rule you created had no effect, which IAM policy a container needs to emit segments, how a trace ID is built and propagated, what the daemon is and when you do not need one, and what AWS now recommends instead of the X-Ray SDK. This guide covers those mechanics.
It assumes you have the Domain 4 overview from the monitoring, troubleshooting and optimization guide, which covers CloudWatch metrics and logs, the CloudWatch vs CloudTrail trap and the one-paragraph X-Ray summary. None of that is re-taught. One framing note before we start: on 25 February 2026 the X-Ray SDKs and the X-Ray daemon entered maintenance mode (security fixes only, no new features) and AWS’s documented recommendation is to instrument with OpenTelemetry, through the AWS Distro for OpenTelemetry (ADOT) or the CloudWatch agent, and send traces to X-Ray. The X-Ray service, its data model, its sampling rules and its console (now inside CloudWatch) are unchanged, and the exam still speaks in X-Ray vocabulary. You need both the classic mechanics and the current recommendation.
1. The Data Model: Segments, Subsegments, Traces
X-Ray receives segments, groups segments sharing a trace ID into traces, and compiles traces into a service graph (the console’s trace map). A segment is a JSON segment document of at most 64 KB describing one service’s work on one request: host, request, response, timing, and any errors. The minimum complete segment is five fields:
{
"name": "example.com",
"id": "70de5b6f19ff9a0a",
"start_time": 1.478293361271E9,
"trace_id": "1-581cf771-a006649127e371903a2de979",
"end_time": 1.478293361449E9
}Facts the exam draws on:
idis a 64-bit identifier in 16 hex digits, unique within the trace.trace_idhas three hyphen-separated parts: the version (1), the original request time in epoch seconds as 8 hex digits, and a globally unique 96-bit random value as 24 hex digits. X-Ray now also accepts W3C Trace Context IDs reformatted this way (4efaaf4d1e8720b39541901950019ee5becomes1-4efaaf4d-1e8720b39541901950019ee5), which is what makes OpenTelemetry interoperable.- Instead of
end_timeyou may sendin_progress: trueto register a long-running request immediately and overwrite it with a complete segment later. One complete segment and zero or one in-progress segment per request. originidentifies the compute (AWS::EC2::Instance,AWS::ECS::Container,AWS::ElasticBeanstalk::Environment); it is set by the SDK plugins and is what the “service type” field in sampling rules matches.- Optional fields:
service.version,user(indexed, searchable),http,aws,annotations,metadata,subsegments, and the error trioerror/throttle/faultwith acauseobject holding exception stacks.
A subsegment is a segment’s view of one downstream call (an AWS SDK call, an HTTP request, a SQL query) or of a block of your own code. It carries a namespace of aws for AWS SDK calls or remote for everything else, and an aws block with operation, table_name, queue_url, request_id for AWS calls. Subsegments can nest, can be embedded in the segment, or can be sent independently (type: subsegment plus trace_id and parent_id), which is how you trace work that continues after the response was sent and how you stay under 64 KB.
Inferred segments are the detail that explains service maps. DynamoDB, S3 and most AWS services do not send segments. X-Ray uses your subsegment for the call to infer a segment and a node for the downstream service. If the downstream service is instrumented (another Lambda function, your own API), its real segment replaces the inferred one; the node then shows the downstream service’s own timing while the edge between the nodes still comes from your subsegment, which includes network time. That is why a Lambda function node can show 40 ms while the edge into it shows 90 ms.
Finally the three error flags, which filter expressions and the console use everywhere:
| Flag | Meaning | Status codes |
|---|---|---|
error | Client error | 4xx |
throttle | Throttled | 429 |
fault | Server fault | 5xx |
They compound: a downstream 429 that makes your service return a 500 sets throttle on the subsegment and fault on the segment.
2. The Trace Header and Parent-Based Sampling
Propagation happens through one HTTP header:
X-Amzn-Trace-Id: Root=1-5759e988-bd862e3fe1be46a994272793;Parent=53995c3f42cd8ad8;Sampled=1Rootis the trace ID.Parentis the segment or subsegment ID of the caller, so the downstream service can attach its segment under the right node.Sampledis the decision:1record,0do not. (A header with noSampledfield, orSampled=?, asks the receiver to decide.)Lineagemay be appended by Lambda and other services for internal loop detection; your code should not use it.
The first X-Ray-integrated service a request reaches adds the header if there is none. API Gateway adds Root= to every incoming request that lacks one. An Application Load Balancer also stamps a trace header on requests it forwards but does not itself send segments to X-Ray, which is a standard distractor. Instrumented HTTP clients (the X-Ray SDK or OpenTelemetry) add the header to outbound calls so the downstream segment can set its parent_id.
The rule that decides most “why was this not traced” questions: sampling is parent-based. The decision is made once, by the first service that handles the request, and every downstream service honours Sampled=1 or Sampled=0 regardless of its own rules. Consequences:
- A sampling rule you create for a downstream service never fires if that service is only ever called by an upstream service that already decided. Put the rule on the root service.
- A downstream Lambda function with active tracing enabled produces traces for sampled upstream requests automatically, and produces nothing for requests the upstream decided not to sample.
- Because the header can come from the client, a user can inject a trace ID or a
Sampled=1. The documentation’s advice is to stripX-Amzn-Trace-Idfrom untrusted inbound requests at the edge if that matters to you, and to generate trace IDs with a secure random source so nobody can predict the next one.
Inside Lambda the header arrives as the environment variable _X_AMZN_TRACE_ID, which the SDKs read instead of an HTTP header.
3. Sampling Rules: Reservoir, Rate, Priority, Quotas
Without sampling, a busy service would record every request and you would pay for all of it. The default behaviour, which the exam quotes verbatim, is: record the first request each second, and 5% of any additional requests. The “one per second” is the reservoir; the “5%” is the fixed rate applied after the reservoir is used up. It guarantees at least one trace per second while a service has traffic, and a representative sample beyond that.
A sampling rule generalises this. Its fields:
| Field | Meaning |
|---|---|
| Rule name | Unique name |
| Priority | 1–9999; rules are evaluated in ascending order and the first match decides |
| Reservoir | Fixed number of matching requests per second to trace before the rate applies; shared by all instances using the rule |
| Rate | Percentage 0–100 of matching requests to trace after the reservoir (0–1 as a fraction when written in a JSON file) |
| Service name | As it appears on the trace map (for API Gateway: api-name/stage) |
| Service type | AWS::EC2::Instance, AWS::ECS::Container, AWS::EKS::Container, AWS::ElasticBeanstalk::Environment, AWS::APIGateway::Stage, AWS::AppSync::GraphQLAPI, AWS::StepFunctions::StateMachine |
| Host, HTTP method, URL path | Request matching; * and ? wildcards |
| Resource ARN | Stage ARN for API Gateway; the SDKs only use rules with * here |
| Attributes | Header-based matching for API Gateway; the SDKs ignore rules that set attributes |
| SamplingRateBoost (optional) | Anomaly-driven boost: MaxRate the rule may rise to during an anomaly and CooldownWindowMinutes between boosts |
Three mechanics matter more than the field list:
Central vs local rules. You can ship a sampling-rules.json with your code, but then every instance applies its reservoir independently, so ten instances record ten traces per second instead of one, and changing a rule means redeploying. Rules defined in the console (now under CloudWatch Settings → X-Ray traces → Sampling rules) are managed by the service: the SDK or ADOT polls GetSamplingRules, reports what it sampled, and receives a per-instance quota from GetSamplingTargets so the reservoir is split evenly across instances. Until a quota arrives, an instance borrows one trace per second if the reservoir is at least 1. “Change sampling without redeploying” and “consistent sampling across an Auto Scaling group” both point at centrally managed rules.
Which services honour rules. API Gateway (REST), AppSync and Step Functions apply your sampling rules natively, and so does code instrumented with the X-Ray SDK or ADOT on EC2, ECS, EKS and Elastic Beanstalk. Lambda does not. Lambda’s active tracing uses the fixed default (one request per second plus 5%) and the Lambda documentation states that you cannot configure the sampling rate for your functions. A question that says “configure a sampling rule so that 100% of invocations of this Lambda function are traced” is describing something that only works if the caller (for example API Gateway) is the root and the rule targets it.
Examples from the documentation, worth recognising:
| Goal | Rule |
|---|---|
| Record almost nothing by default | Default rule with reservoir 0, rate 5 |
| Debug one route | Priority 1, reservoir 1, rate 100, method PUT, path /history/* |
| Guarantee more coverage of writes | Priority 100, reservoir 10, rate 10, method POST |
| Trace everything (disable sampling) | A high-priority catch-all rule with rate 100 |
Quota: 25 custom sampling rules per Region (adjustable). Statistics per rule (matched, sampled by rate, sampled by reservoir, borrowed) appear on the Sampling page in 10-second windows.
4. Getting Segments to X-Ray: SDK, Daemon, ADOT, CloudWatch Agent
The classic path has three hops. Your code, instrumented with an X-Ray SDK, builds segment documents and sends them as UDP datagrams to port 2000 on 127.0.0.1 (or wherever AWS_XRAY_DAEMON_ADDRESS points). The X-Ray daemon listens there, buffers segments, and uploads them in batches with the PutTraceSegments API using whatever AWS credentials it finds (instance profile, task role, environment). The daemon also proxies the sampling API calls (GetSamplingRules, GetSamplingTargets) on TCP 2000.
Where the daemon runs, per compute service:
| Compute | How the daemon gets there | Credentials |
|---|---|---|
| EC2 | You install and run it (user data, RPM/DEB, or the executable) | Instance profile with AWSXRayDaemonWriteAccess |
| ECS / Fargate | A sidecar container in the task definition (public.ecr.aws/xray/aws-xray-daemon:3.x or amazon/aws-xray-daemon) exposing 2000/udp; the app sets AWS_XRAY_DAEMON_ADDRESS to the sidecar (localhost:2000 in awsvpc mode) | Task role with AWSXRayDaemonWriteAccess |
| Elastic Beanstalk | Built in: set the XRayEnabled option (aws:elasticbeanstalk:xray namespace in .ebextensions, or the console) | The environment’s instance profile |
| Lambda | Lambda runs it for you when active tracing is on; nothing to install | Execution role with AWSXRayDaemonWriteAccess |
| EKS | Daemon or ADOT collector as a sidecar or DaemonSet | Pod identity / IRSA |
The managed policy AWSXRayDaemonWriteAccess grants xray:PutTraceSegments, xray:PutTelemetryRecords, xray:GetSamplingRules, xray:GetSamplingTargets and xray:GetSamplingStatisticSummaries, which is everything an emitter needs; AWSXRayWriteOnlyAccess is the narrower variant. Read-side policies (AWSXRayReadOnlyAccess) are for people and dashboards, not for applications.
Now the 2026 reality. The X-Ray SDKs (Java, Node.js, Python, .NET, Go, Ruby) and the daemon are in maintenance mode as of 25 February 2026: security fixes only. The documentation recommends two replacements:
- OpenTelemetry via ADOT: the AWS Distro for OpenTelemetry SDKs and auto-instrumentation agents produce OTLP spans; the ADOT Collector (sidecar, DaemonSet or Lambda layer) exports them to X-Ray, converting W3C trace IDs into X-Ray format. ADOT supports X-Ray remote sampling, so your console sampling rules still apply.
- The CloudWatch agent (version 1.300025.0 and later) can receive traces from either OpenTelemetry or X-Ray SDK clients and forward them to X-Ray, so one agent collects metrics, logs and traces. The documented benefit is fewer agents to run.
For exam purposes: if a question is written in classic vocabulary (daemon, SDK, UDP 2000), answer in classic vocabulary. If it asks for the recommended, least-operational-overhead or vendor-neutral path, the answer is OpenTelemetry with ADOT or the CloudWatch agent, still landing in X-Ray.
5. Instrumenting Code: What the SDK Does for You
An X-Ray SDK does four things: creates the segment for each incoming request (web framework middleware), patches AWS SDK clients, HTTP clients and database drivers so each downstream call becomes a subsegment, exposes an API for custom subsegments, annotations and metadata, and makes the sampling decision using the rules. Python, as a representative example:
from aws_xray_sdk.core import xray_recorder, patch_all
from aws_xray_sdk.ext.flask.middleware import XRayMiddleware
patch_all() # boto3, requests, sqlite3, psycopg2, ...
xray_recorder.configure(
service='orders-api', # segment name / service name
plugins=('EC2Plugin', 'ECSPlugin'), # fill origin + aws block
sampling=True,
context_missing='LOG_ERROR', # don't crash outside a segment
)
XRayMiddleware(app, xray_recorder) # one segment per request
@app.route('/orders/<order_id>')
def get_order(order_id):
with xray_recorder.in_subsegment('load-order') as sub:
sub.put_annotation('order_id', order_id) # indexed
sub.put_metadata('debug', {'cache': 'miss'}) # not indexed
return table.get_item(Key={'id': order_id}) # auto subsegment (namespace aws)Environment variables worth knowing because they appear as answers: AWS_XRAY_DAEMON_ADDRESS (where to send UDP), AWS_XRAY_TRACING_NAME (segment name override), AWS_XRAY_CONTEXT_MISSING (RUNTIME_ERROR throws when you record outside a segment, LOG_ERROR logs instead; LOG_ERROR is what you want for background threads and local tests), and the plugins (EC2Plugin, ECSPlugin, ElasticBeanstalkPlugin) that set origin and the aws resource block so sampling rules can match on service type.
The OpenTelemetry equivalent uses the OTel SDK plus the ADOT auto-instrumentation agent for your language; a span maps to an X-Ray segment or subsegment, span attributes become annotations or metadata (ADOT lets you choose which attribute keys are indexed as annotations), and the traceparent header replaces X-Amzn-Trace-Id between OTel-instrumented services while the collector bridges the two formats.
6. Annotations vs Metadata, Precisely
| Annotations | Metadata | |
|---|---|---|
| Indexed / searchable | Yes: filter expressions, groups, GetTraceSummaries | No: visible only in the full segment via BatchGetTraces or the trace detail view |
| Value types | String, number, boolean only | Anything JSON, including objects and arrays |
| Key rules | Alphanumeric and underscore (dots must be bracketed in filters) | Any; keys starting AWS. are reserved for AWS SDKs |
| Limit | 50 indexed annotations per trace | Bounded only by the 64 KB segment document |
| Typical content | Customer ID, order ID, feature flag, region, tenant | Request payloads, debug state, config dumps |
Three nuances beyond the table:
- Annotations and metadata can be put on any segment or subsegment and are aggregated at the trace level for search. On Lambda you cannot modify the function segment Lambda itself creates, so annotations go on a subsegment you open inside the handler; they are still searchable at trace level.
- The segment’s
userfield is indexed too and has its own filter keyword (user = "alice"), so “search traces by user ID” has two correct implementations: setuseron the segment, or add an annotation. - Over 50 annotations in a trace: the excess is stored but not indexed. If a search cannot find a trace you know exists, count the annotations.
7. Filter Expressions and Groups
Filter expressions are how you find traces, and the console builds them for you when you click a node. The grammar is keyword operator value, combined with AND / OR, with complex keywords that apply a nested expression to a downstream service or edge.
| Keyword family | Examples |
|---|---|
| Boolean | ok, error, throttle, fault, partial, inferred, root; negate with ! |
| Number | responsetime > 5, duration >= 5 AND duration <= 8, http.status != 200 |
| String | http.url CONTAINS "/api/game/", http.method = "POST", user = "alice", http.clientip, resource.arn, availabilityzone |
| Annotation | annotation[gameid] = "817DL6VO", annotation[age] > 29, !annotation[age] (not set) |
| Service / edge | service("api.example.com") { fault }, service() { fault } (anywhere downstream), edge("api", "backend") { error } |
| Identity | service(id(name: "random-name", type: "AWS::Lambda::Function")) { error } to separate a Lambda function node from the Lambda service node of the same name |
| Group | group.name = "high_response_time" AND user = "alice" |
The distinction worth remembering is between responsetime (the time the server took to respond) and duration (the whole request including downstream calls), and that simple keywords only look at the trace’s root: a downstream fault your code handled and turned into a 200 is found only with service() { fault }.
A group is a saved filter expression. Incoming traces that match it get their own service map, trace summaries and CloudWatch metrics published every minute, which is how you alarm on “fault rate for checkout traces” rather than for a whole service. Quota: 25 groups per account per Region. Changing a group’s expression affects only new traces. Groups are billed by retrieved traces that match. X-Ray Insights, enabled per group, runs anomaly detection on the group’s fault and response-time metrics and can send notifications through EventBridge.
Retention: traces and the service graph are kept for 30 days, and recorded data can be modified (for example, by late-arriving subsegments) for 7 days.
8. Lambda: Active vs PassThrough, Fixed Sampling, Two Segments
Lambda’s tracing configuration has two modes:
PassThrough(the default): Lambda forwards the trace header to downstream calls your code makes but does not send traces, even when the incoming header saysSampled=1. If no header arrives, Lambda generates one with the decision not to sample. The documentation notes this is a change from older behaviour where Lambda traced automatically whenever API Gateway said so; if you depend on that, switch to Active.Active: Lambda runs the daemon, records segments for sampled invocations and sends them. For requests with an upstream decision, Lambda honours it. For requests with no decision (a schedule, an uninstrumented caller), Lambda samples at the fixed 1 request per second plus 5%, which you cannot change.
Enabling it: console toggle, aws lambda update-function-configuration --function-name my-function --tracing-config Mode=Active, CloudFormation TracingConfig: {Mode: Active} on AWS::Lambda::Function, or SAM Tracing: Active on AWS::Serverless::Function (the SAM guide covers the template side). Tracing mode is version-specific configuration: it is frozen when you publish a version and cannot be changed on that version afterwards, only on $LATEST and future versions. The execution role needs xray:PutTraceSegments and xray:PutTelemetryRecords, which AWSXRayDaemonWriteAccess provides; the console adds them for you when you enable tracing there.
What Lambda records: two segments, two nodes, both named after the function but with different origins. The AWS::Lambda segment is the Lambda service preparing the execution environment (scheduling the MicroVM, creating or unfreezing it, downloading code and layers). The AWS::Lambda::Function segment is your function’s work. An error on the first means Lambda had a problem; on the second, your code did. In the long-standing structure the function segment carries Initialization, Invocation and Overhead subsegments (plus Restore for SnapStart functions, see the SnapStart guide), mapping directly onto the execution-environment lifecycle from the versions, aliases and concurrency guide. Lambda is rolling out a new structure in which the Invocation subsegment disappears, your subsegments attach directly to the function segment, and the function segment carries annotations such as aws.responseLatency, aws.responseDuration, aws.runtimeOverhead and aws.extensionOverhead. Either way, your code cannot modify the function segment: you add subsegments, annotations and metadata under it, and you cannot record work outside the handler’s scope.
Two constraints that show up in questions: a downstream Lambda function only produces traces if the root function that invokes it also has tracing enabled; and X-Ray tracing is not supported for event source mappings from Amazon MSK, self-managed Kafka, Amazon MQ (ActiveMQ and RabbitMQ) or DocumentDB. A visible gap between Initialization and Invocation is normal for provisioned concurrency and for proactive initialisation, not a bug.
For instrumentation inside the function you have the X-Ray SDK bundled with your code (maintenance mode), the ADOT Lambda layer (OpenTelemetry, with the collector in the layer), or Powertools for AWS Lambda’s Tracer utility, which wraps the X-Ray SDK with decorators for handlers and methods.
9. API Gateway: REST Only, Per Stage, Honours Rules
API Gateway integration facts, each of which has been an exam answer:
- X-Ray tracing is supported for REST APIs only, on all endpoint types (Regional, edge-optimized, private). HTTP APIs and WebSocket APIs are not supported.
- It is enabled per stage (Logs/Tracing tab, “Enable X-Ray Tracing”, then redeploy; or
aws apigateway update-stage --rest-api-id <id> --stage-name prod --patch-operations op=replace,path=/tracingEnabled,value=true). - API Gateway adds the trace header to every incoming request that does not have one, and with active tracing it is usually the root that makes the sampling decision using your sampling rules, matching on service name
api-name/stage, service typeAWS::APIGateway::Stage, the stage ARN as resource ARN, and request headers as attributes. - With active tracing disabled, a stage still records a segment when an upstream instrumented client already sampled the request.
- With HTTP integrations, each distinct request URL path becomes its own service node on the trace map, which can blow through the 10,000-node trace-map limit; the documented mitigation is to move variable values into query strings or request bodies.
Integration details for REST APIs are in the API Gateway guide.
10. Event-Driven Tracing: SQS → Lambda Trace Linking
Traces are request-shaped, and a queue breaks the request. X-Ray’s answer is trace linking. When an instrumented producer sends to SQS, the trace header travels on the message as the AWSTraceHeader system attribute. When Lambda consumes a batch, each message’s trace is linked to the consumer’s trace rather than merged into it: the trace map shows a dashed-line edge from the SQS node to the Lambda consumer, selecting it shows a received event age histogram, and the trace detail page shows upstream and downstream linked traces as separate boxes.
Limits: each segment can link to 20 traces and a trace can hold 100 links; a function that sends many messages per invocation can exceed the trace document size (100 KB lower limit, 500 KB dynamic upper limit), and the SDKs document an oversampling mitigation for that case. Step Functions and AppSync also support active tracing with sampling rules; SNS and EventBridge propagate the header to subscribers and targets. The broader event-driven patterns are in the SQS, SNS, EventBridge and Step Functions guide.
11. X-Ray Inside CloudWatch: Transaction Search and Application Signals
The X-Ray console now lives in CloudWatch (Traces, Trace Map, Settings). Two newer capabilities sit on top of it and are increasingly the “modern” answer:
- Transaction Search ingests 100% of spans sent to X-Ray as structured logs into the CloudWatch Logs log group
aws/spans(X-Ray segments are converted to the OpenTelemetry semantic-convention format with W3C trace IDs), and indexes a configurable percentage of them as X-Ray trace summaries for search and analytics (1% when you first enable it, adjustable upward). You get complete traces (up to 10,000 spans) with no broken traces from sampling, Logs Insights-style free-form queries over span attributes, metric filters, subscription filters and data masking on span data. Pricing follows CloudWatch’s model (ingestion plus indexing) rather than X-Ray’s per-trace pricing. - Application Signals builds an application-centric APM view (services, dependencies, SLOs, dashboards) from the same spans, with a pre-packaged OpenTelemetry setup through ADOT or the CloudWatch agent.
For the exam, recognise the vocabulary: “spans”, “aws/spans”, “100% of spans without sampling”, “OpenTelemetry endpoint”, “Application Signals” all point here; “segments”, “daemon”, “sampling rules”, “annotations” point at classic X-Ray. Both are one service viewed from two eras.
12. Quotas and Numbers That Appear in Questions
| Quota | Value |
|---|---|
| Segment document size | 64 KB (not adjustable) |
| Indexed annotations per trace | 50 (not adjustable) |
| Custom sampling rules per Region | 25 (adjustable) |
| Groups per account per Region | 25 |
| Segments per second | 2,600 |
| Trace and service-graph retention | 30 days |
| Trace data modification period | 7 days |
| Trace document size | 100 KB lower limit, 500 KB dynamic upper limit |
| Trace-map nodes | 10,000 |
| Links per segment / per trace | 20 / 100 |
| Daemon port | UDP 2000 (TCP 2000 for sampling API proxy) |
| Default sampling | 1 request/second reservoir + 5% rate |
| Lambda sampling (Active, no upstream decision) | 1/second + 5%, not configurable |
Pricing shape, in case a cost-optimisation question appears: X-Ray charges per trace recorded and per trace retrieved or scanned, with a perpetual free tier of 100,000 traces recorded and 1,000,000 retrieved or scanned per month. Sampling is the cost lever; groups add retrieval cost; Transaction Search is billed through CloudWatch.
13. Scenario Drill
| Scenario | Answer | Why |
|---|---|---|
| API Gateway traces appear but the Lambda behind it shows no segment | Enable Active tracing on the function | PassThrough never sends traces, even with Sampled=1 |
| Need 100% of a Lambda function’s invocations traced | Set a 100% rule on the root (API Gateway) or accept Lambda’s fixed rate | Lambda does not apply custom sampling rules |
Sampling rule created for payments-service has no effect | It is always called by an upstream root that already decided; put the rule on the root | Parent-based sampling |
| Change sampling for an ASG fleet without redeploying, and keep total traces/second constant | Centrally managed sampling rules in the console | Service assigns quotas per instance; local JSON reservoirs add up |
| Find all traces for customer 12345 | Record it as an annotation (or the user field) and filter with annotation[customer_id] = "12345" | Metadata is not indexed |
| Store the full request payload with the trace | Metadata | Any JSON, not indexed, bounded by 64 KB |
| ECS task on Fargate shows no traces | Add the X-Ray daemon (or ADOT collector) sidecar, point the app at it, give the task role AWSXRayDaemonWriteAccess | No daemon, or no permission, no segments |
EC2 app gets AccessDenied on PutTraceSegments | Attach AWSXRayDaemonWriteAccess to the instance profile | The daemon uses the instance credentials |
| Elastic Beanstalk app, least effort to enable tracing | XRayEnabled: true in .ebextensions or console | Platform runs the daemon |
| Which service returned the 500? | Trace map / service() { fault }; node with fault | Flags propagate per segment |
| Downstream DynamoDB node shows latency but DynamoDB “does not send traces” | It is an inferred segment from your subsegment | Edges and inferred nodes come from the client’s view |
Lambda trace shows a long AWS::Lambda segment, short function segment | Cold start / environment preparation, not your code | Two segments, two origins |
| HTTP API (not REST) needs X-Ray | Not supported on API Gateway HTTP APIs; trace in the integration (Lambda Active) instead | REST only |
| Messages pass through SQS and the consumer’s trace is disconnected | Expected: traces are linked, shown with a dashed edge, not merged | Trace linking semantics |
Searching by order_id finds nothing although it is annotated | Trace exceeded 50 indexed annotations or value is nested (object) | Only scalars, first 50, indexed |
| Vendor-neutral instrumentation that still lands in X-Ray with least operational overhead | OpenTelemetry (ADOT) or the CloudWatch agent, not the X-Ray SDK | SDK and daemon in maintenance mode since Feb 2026 |
| Need every span, no sampling gaps, and ad-hoc analytics over span attributes | Enable Transaction Search | 100% of spans into aws/spans |
Common Mistakes
- Assuming Lambda traces when API Gateway does. PassThrough is the default and sends nothing. Turn on Active.
- Writing sampling rules for a downstream service. Only the root decides. Rules apply where the decision is made.
- Trying to configure Lambda’s sampling rate. It is fixed at 1/second + 5% for requests without an upstream decision.
- Putting searchable values in metadata. Only annotations (and
user) are indexed, and only the first 50 annotations per trace. - Using objects as annotation values. Annotations are strings, numbers and booleans only.
- Forgetting the daemon on ECS and EC2. The SDK speaks UDP to port 2000; without a listener, segments vanish silently.
- Granting the wrong principal. The daemon’s credentials are the instance profile, task role or execution role, not the user running the deployment.
- Expecting X-Ray on HTTP APIs. REST APIs only, per stage, redeploy after enabling.
- Expecting SQS to merge traces. Producer and consumer traces are linked, not joined.
- Treating the X-Ray SDK as the current recommendation. It is in maintenance mode; the documented path is OpenTelemetry via ADOT or the CloudWatch agent, still exporting to X-Ray.
Practising for the DVA-C02
X-Ray questions are short but precise, and they punish approximate knowledge: PassThrough vs Active, root vs downstream, annotation vs metadata, REST vs HTTP API. The quickest way to make these reflexive is to trace a real three-hop application (API Gateway → Lambda → DynamoDB, then add an SQS hop) with Active tracing and read the raw segment documents with aws xray batch-get-traces, so the JSON in section 1 stops being abstract. Then drill the recognition. Sailor.sh’s AWS Certified Developer - Associate (DVA-C02) Mock Exam Bundle includes Domain 4 scenarios across X-Ray sampling, Lambda tracing modes, container and EC2 daemon setup, annotation-vs-metadata and the OpenTelemetry migration framing, each with an explanation that names the mechanism so a near-miss becomes a lesson rather than a lucky guess. The free DVA-C02 practice questions are a good warm-up, and the DVA-C02 exam guide covers how Domain 4 sits alongside the rest of the blueprint.
Frequently Asked Questions
What is in the X-Amzn-Trace-Id header?
Root (the trace ID in 1-<8 hex epoch seconds>-<24 hex random> format), optionally Parent (the calling segment or subsegment ID) and Sampled (1 or 0). Lambda and some services append a Lineage field you should ignore. The first X-Ray-integrated service adds the header; instrumented clients propagate it.
How does X-Ray sampling work by default?
The first request each second is recorded (the reservoir) and 5% of additional requests (the rate). The decision is made once by the root service and honoured downstream. Centrally managed sampling rules let you change this per service, path, method or host without redeploying; Lambda’s own sampling is fixed.
What is the difference between annotations and metadata?
Annotations are scalar key-value pairs that X-Ray indexes for filter expressions and groups, up to 50 per trace. Metadata is arbitrary JSON that is stored with the trace but not indexed and cannot be searched.
Does Lambda trace automatically when API Gateway tracing is enabled?
No. In the default PassThrough mode Lambda forwards the header but sends no traces. Enable Active tracing on the function; it then honours the upstream sampling decision and samples uninstrumented invocations at 1/second plus 5%.
Do I need the X-Ray daemon on Lambda?
No. Lambda runs the daemon when active tracing is on. You need it (or the ADOT collector or the CloudWatch agent) on EC2, ECS and EKS, and Elastic Beanstalk runs it for you when XRayEnabled is set.
Why did a sampling rule I created not change anything?
Either the service it targets is downstream of a root that already made the decision, or the service does not apply custom rules (Lambda), or the rule’s priority is lower (a higher number) than another rule that matched first.
Is the X-Ray SDK deprecated?
The X-Ray SDKs and daemon entered maintenance mode on 25 February 2026 and receive security fixes only. AWS recommends migrating to OpenTelemetry (ADOT or the CloudWatch agent) while still sending traces to X-Ray. The X-Ray service itself is not deprecated.
Which API Gateway API types support X-Ray?
REST APIs only, enabled per stage. HTTP APIs and WebSocket APIs do not support X-Ray tracing; trace inside the integration instead.
Conclusion
X-Ray is a small set of mechanisms that combine into everything the console shows. Segments and subsegments are JSON documents with a strict shape and a 64 KB ceiling; a trace is the set sharing a Root ID; a trace header carries that ID plus a sampling decision made once at the root and honoured everywhere downstream; sampling rules with reservoirs, rates and priorities decide what is recorded and are applied where the decision is made; annotations are the fifty searchable fields and metadata is the rest; a daemon, an ADOT collector or the CloudWatch agent moves segments from UDP 2000 to PutTraceSegments under whatever IAM identity the compute has.
Layer the service specifics on top (Lambda’s PassThrough default and fixed sampling, API Gateway’s REST-only per-stage switch, SQS trace linking) and the February 2026 shift toward OpenTelemetry, and the Domain 4 questions stop being trivia. Each one is asking which mechanism failed to fire, and once you can name the mechanism, the answer is the one that fixes it.