The DVA-C02 messaging story doesn’t end with queues and topics. When a scenario says real-time, multiple consumers, ordered, or replay, the answer usually isn’t SQS or SNS — it’s Amazon Kinesis Data Streams. Streaming shows up in the Development domain as a producer/consumer coding problem and in Troubleshooting as a throttling or “consumer falling behind” scenario, so knowing where Kinesis beats a queue is worth real points.
This guide is the developer-altitude companion to the DVA-C02 event-driven services guide, which covers SQS, SNS, EventBridge, and Step Functions. Here we go deep on streaming: how shards and partition keys work, how to write and read records, the two consumer models, and — most importantly for the exam — exactly when to reach for Kinesis instead of a queue.
The Kinesis Family: Know Which Service Is Being Described
“Kinesis” is a family, and the exam tests whether you can tell the members apart from a one-line description:
| Service | What it does | Managed for you? | Replay? |
|---|---|---|---|
| Kinesis Data Streams (KDS) | Ingest and store a real-time stream you build consumers for | You manage shards / capacity | Yes, within retention |
| Kinesis Data Firehose | Deliver streaming data to S3, Redshift, OpenSearch, Splunk | Fully managed, no shards | No |
| Kinesis Data Analytics | Run SQL / Flink over a stream in real time | Fully managed | N/A |
Firehose vs Data Streams is the most common family question. If the scenario wants streaming data loaded into S3 or Redshift with the least code and no consumer to manage, that’s Firehose — it buffers (by size or time) and delivers, and can transform records with a Lambda in flight, but it cannot replay. If the scenario needs sub-second custom processing, multiple independent consumers, or the ability to re-read old records, that’s Data Streams. The rest of this guide is about Data Streams.
Shards: the Unit of Capacity
A stream is made of shards, and a shard is a fixed unit of throughput. Memorize these numbers — the exam builds throttling questions directly on them:
- Ingress (writes): 1 MB/sec or 1,000 records/sec per shard, whichever you hit first.
- Egress (reads, shared fan-out): 2 MB/sec per shard, shared across all consumers.
- Record size: up to 1 MB per record.
Total stream capacity is those limits multiplied by the shard count. A 4-shard stream ingests up to 4 MB/sec or 4,000 records/sec. Exceed a shard’s write limit and the producer gets a ProvisionedThroughputExceededException — the single most important error name for this topic.
Provisioned vs On-Demand Capacity
Data Streams has two capacity modes, and the choice is a favourite exam discriminator:
- Provisioned mode — you specify the shard count and pay per shard-hour. You scale by resharding (splitting/merging shards). Best when throughput is predictable and you want to control cost.
- On-demand mode — Kinesis manages capacity automatically and you pay per throughput (GB written/read). Best for unknown or spiky workloads where you don’t want to manage shards.
Exam phrasing: “unpredictable traffic, don’t want to manage capacity” → on-demand. “Steady, well-understood throughput, cost-sensitive” → provisioned.
Partition Keys: Ordering and the Hot-Shard Trap
Every record you write carries a partition key. Kinesis takes the MD5 hash of that key and maps it to a shard. Two consequences flow from this, and both are tested:
-
Ordering is per shard. All records with the same partition key land on the same shard and are read in the order they were written. There is no global ordering across shards. If order matters for a given entity — say, all events for one
userId— use that entity’s id as the partition key so its records stay on one shard, in order. -
A skewed partition key creates a hot shard. If 90% of your records share one partition key (or a handful), they all hash to the same shard and throttle it, even though the stream’s total capacity looks fine. This is the streaming twin of the DynamoDB hot-partition problem covered in the DVA-C02 DynamoDB guide.
Exam signal: “Some records are throttled with ProvisionedThroughputExceededException while overall stream throughput is below the provisioned limit.” That’s a hot shard from a poorly-chosen partition key. The fix is a higher-cardinality, evenly-distributed partition key — not simply adding shards (adding shards won’t help if everything still hashes to one).
Producers: Getting Records In
You have several ways to write to a stream, and the exam expects you to pick the right one.
SDK — PutRecord and PutRecords. PutRecord writes one record; PutRecords writes a batch in a single call for far higher throughput. The critical detail: PutRecords can partially fail. The call returns HTTP 200 even when some records were rejected, so you must inspect FailedRecordCount and the per-record ErrorCode, then retry only the failed records (typically ProvisionedThroughputExceededException on a hot shard).
import boto3
kinesis = boto3.client("kinesis")
resp = kinesis.put_records(
StreamName="orders",
Records=[
{"Data": b'{"orderId":"A1"}', "PartitionKey": "customer-42"},
{"Data": b'{"orderId":"A2"}', "PartitionKey": "customer-42"},
],
)
# A 200 response does NOT mean every record succeeded
if resp["FailedRecordCount"] > 0:
for i, r in enumerate(resp["Records"]):
if "ErrorCode" in r:
print("retry record", i, r["ErrorCode"]) # e.g. ProvisionedThroughputExceededExceptionKinesis Producer Library (KPL). A higher-level library that batches and aggregates many small user records into fewer, larger Kinesis records for better throughput and cost. The trade-off is latency: KPL buffers records for up to RecordMaxBufferedTime, so it’s for high-throughput producers that can tolerate a little delay. Aggregated records are de-aggregated by the KCL on the consumer side.
Kinesis Agent. A stand-alone Java app that tails log files and ships them to a stream — no code required. Handy for “collect logs from EC2 without writing a producer” scenarios.
Retry & backoff. For any producer, ProvisionedThroughputExceededException should be handled with exponential backoff and retry. Persistent throttling means fix the partition key or add capacity.
Consumers: Shared Fan-Out vs Enhanced Fan-Out
This is the highest-value comparison in the whole topic. There are two ways to read a stream, and choosing the wrong one is a classic wrong answer.
Shared (classic) fan-out — GetRecords. Consumers pull records. The catch: the 2 MB/sec egress per shard is shared across every consumer of that shard, and each shard allows only 5 GetRecords calls per second. Two consumers on the same shard each effectively get ~1 MB/sec, and adding more increases read contention and latency. Fine for one or two consumers; it doesn’t scale to many.
Enhanced fan-out (EFO) — SubscribeToShard. Each registered consumer gets its own dedicated 2 MB/sec per shard pipe, and records are pushed over HTTP/2 with ~70 ms latency. This is the answer when you have multiple consumers that each need full throughput or need low latency. It costs more, so don’t reach for it when one consumer will do.
| Need | Choose |
|---|---|
| One or two consumers, cost-sensitive | Shared fan-out (GetRecords) |
| Many consumers, each needs full throughput | Enhanced fan-out |
| Lowest latency (~70 ms), push-based | Enhanced fan-out |
Lambda as a consumer. Lambda reads Kinesis via an event source mapping that polls shards and invokes your function with batches. Key knobs the exam likes: by default one Lambda invocation processes one shard at a time in order; parallelization factor lets multiple invocations process one shard concurrently (while preserving per-partition-key order); and on-failure destinations, BisectBatchOnFunctionError, and MaximumRetryAttempts control error handling so a single poison-pill record doesn’t block the shard forever. You can also subscribe Lambda with enhanced fan-out for lower latency.
Kinesis Client Library (KCL). For custom consumer apps on EC2/containers, the KCL handles the hard parts: it distributes shards across worker instances (one lease per shard), checkpoints progress in a DynamoDB table, and reassigns shards when workers join, leave, or the stream reshards. Two exam-worthy gotchas: the KCL needs IAM permissions for that DynamoDB table (and it counts toward your DynamoDB cost/capacity), and running more KCL workers than shards leaves the extra workers idle — parallelism is capped by shard count.
Resharding: Scaling a Provisioned Stream
In provisioned mode you scale throughput by changing shard count:
- Shard split — one shard becomes two, increasing capacity (use it to relieve a hot shard).
- Shard merge — two shards become one, reducing capacity and cost.
Resharding creates parent/child shard relationships; consumers (and the KCL) must finish reading a parent shard before its children to preserve ordering. You reshard one step at a time, which is why on-demand mode is attractive when you’d otherwise be constantly resizing.
Kinesis Data Streams vs SQS: the Decision the Exam Loves
If you remember one table from this article, make it this one. SQS and Kinesis both move data between components, but their semantics are fundamentally different:
| Dimension | SQS | Kinesis Data Streams |
|---|---|---|
| Model | Queue — each message consumed once, then deleted | Stream — records read by many consumers, retained |
| Consumers | One logical consumer group | Multiple independent consumers, same data |
| Replay | No (message gone after delete) | Yes, re-read within retention (24 h default, up to 365 days) |
| Ordering | Only with FIFO queues | Per shard (per partition key) |
| Scaling | Fully elastic, no capacity to manage | Shards (provisioned) or on-demand |
| Best for | Decoupling, work queues, buffering tasks | Real-time analytics, multiple consumers, ordered event streams |
Decision rules the exam rewards:
- “Multiple applications must each process the same events in real time” → Kinesis (SQS deletes a message once it’s consumed, so a second app never sees it).
- “Need to replay the last few hours of events after a bug fix” → Kinesis (SQS has no replay).
- “Simply decouple a producer from a pool of workers, each task handled once” → SQS.
- “Preserve order of a high-throughput event stream per entity” → Kinesis with a per-entity partition key (SQS FIFO caps at lower throughput).
Encryption, Retention, and Monitoring
- Encryption: server-side encryption with KMS at rest, HTTPS/TLS in transit. A “encrypt streaming data at rest” question points to SSE-KMS on the stream.
- Retention: records live 24 hours by default, extendable up to 365 days — this is what makes replay possible.
- Monitoring: the CloudWatch metric to know is
GetRecords.IteratorAgeMilliseconds. A rising iterator age means consumers are falling behind the stream — the classic troubleshooting signal. Fixes: add shards (or switch to on-demand), use enhanced fan-out, raise Lambda’s parallelization factor, or add consumer capacity. Also watchWriteProvisionedThroughputExceededfor producer-side throttling. Instrument end-to-end latency with X-Ray as covered in the DVA-C02 monitoring & optimization guide.
Exam Scenario Cheat Sheet
| Scenario clue | Answer |
|---|---|
| Multiple apps consume the same real-time events | Kinesis Data Streams |
| Load streaming data into S3/Redshift, least code | Kinesis Data Firehose |
| Records throttled but total throughput is low | Hot shard → better partition key |
| Many consumers each need full throughput / low latency | Enhanced fan-out |
| One or two consumers, minimize cost | Shared fan-out (GetRecords) |
PutRecords returns 200 but some data missing | Check FailedRecordCount, retry failed records |
Consumers falling behind (high IteratorAgeMilliseconds) | Add shards / EFO / parallelization factor |
| Custom consumer that must track progress & handle resharding | KCL (checkpoints in DynamoDB) |
| Unpredictable, spiky throughput, no shard management | On-demand capacity mode |
| High-throughput producer of many small records | KPL (aggregation) |
Practice With Realistic Streaming Questions
Kinesis questions reward candidates who can instantly translate a scenario clue into the right service and setting — “multiple consumers, same data” to Data Streams, “high iterator age” to a scaling fix, “200 with missing records” to a PutRecords partial-failure retry. That recognition comes from repetition against exam-style questions, not from re-reading docs.
The AWS Certified Developer Associate (DVA-C02) Mock Exam Bundle includes eight full-length exams — 520+ questions across all four DVA-C02 domains — with streaming and messaging scenarios and explanations that reinforce why each answer is right. Pair it with the DVA-C02 study plan to slot streaming in alongside the rest of the Development domain, review the exam topics breakdown to see how messaging fits the whole blueprint, and revisit the event-driven services guide so the Kinesis-vs-SQS boundary is second nature.
Frequently Asked Questions
When should I use Kinesis Data Streams instead of SQS?
Use Kinesis when multiple independent consumers must each process the same records, when you need to replay data within a retention window, or when you need ordered, high-throughput streaming. Use SQS when you just need to decouple a producer from workers and each message should be processed once and then deleted.
What is the difference between shared fan-out and enhanced fan-out?
Shared (classic) fan-out shares each shard’s 2 MB/sec read throughput across all consumers and uses pull-based GetRecords. Enhanced fan-out gives each registered consumer its own dedicated 2 MB/sec per shard with push-based SubscribeToShard and ~70 ms latency. Use enhanced fan-out when you have many consumers or need low latency; it costs more.
Why are my records throttled when the stream isn’t at capacity?
Almost always a hot shard. Records are mapped to shards by hashing the partition key, so a low-cardinality or skewed partition key sends most traffic to one shard, throttling it while the rest of the stream sits idle. Fix it with a higher-cardinality, evenly distributed partition key.
Does PutRecords guarantee all records are written?
No. PutRecords can partially fail and still return an HTTP 200 response. You must check FailedRecordCount and each record’s ErrorCode, then retry only the failed records, typically with exponential backoff.
What does the KCL use DynamoDB for?
The Kinesis Client Library stores shard leases and checkpoints in a DynamoDB table so multiple worker instances can coordinate which shard each is processing and resume from the last processed record after a restart or reshard. Your consumer’s IAM role needs access to that table.
Firehose or Data Streams for loading data into S3?
Firehose if you just need managed, near-real-time delivery to S3 (or Redshift/OpenSearch/Splunk) with optional Lambda transformation and no consumer code. Data Streams if you need sub-second processing, multiple independent consumers, or replay — then you build and run the consumer yourself.
Conclusion
Kinesis Data Streams is where the DVA-C02’s messaging knowledge gets tested at its sharpest: shards and partition keys decide throughput and ordering, PutRecords demands partial-failure handling, and the shared-vs-enhanced fan-out choice separates candidates who memorized services from those who understand them. Anchor everything to the two decisions the exam returns to again and again — Kinesis vs SQS (multiple consumers, ordering, replay) and shared vs enhanced fan-out (how many consumers, how much latency) — and back it with the throttling reflexes (ProvisionedThroughputExceededException from a hot shard, rising IteratorAgeMilliseconds from a slow consumer). Get those cold, practice against realistic questions, and streaming becomes some of the most reliable points on the exam.