Amazon Kinesis Data Streams
Easily stream data at any scale
What is Kinesis Data Streams?
Amazon Kinesis Data Streams is a serverless data streaming service that ingests data once, fans it out to multiple consumer applications for real-time processing, and delivers the same data to data lakes for later analysis. Continuously collect gigabytes of data per second from hundreds of thousands of sources, such as clickstreams, application logs, IoT telemetry, orders, and transactions, and make it available in milliseconds. Consumer applications read records from the stream for stream processing or receive the same records through a queue for independent record processing. Managed data delivery writes records within minutes to streaming tables or Amazon S3, where analysts and analytical engines can use them.
Benefits
-
With Amazon Kinesis Data Streams, there are no servers to manage. The on-demand mode eliminates the need to provision or manage capacity required for running applications.
Adjust your capacity to stream gigabytes per second of data with Kinesis Data Streams. Seamlessly scale your capacity up or down to stream gigabytes per second of data with Kinesis Data Streams. Get automatic provisioning and scaling with On-Demand Advantage mode
Multiple applications can consume the same records from a stream at the same time. Stream consumers process records in sequence for work such as analytics and anomaly detection. Queue consumers, powered by Amazon SQS, process each record independently for work such as fulfilling orders and running inference on individual records.
Deliver Kinesis data directly to streaming tables in Amazon S3 Tables or general purpose S3. Eliminate pipeline infrastructure overhead and reduce end-to-end ingestion costs by up to 50% and lower downstream query costs by up to 30% through intelligent inline data compaction.
-
Use built-in integrations with other AWS services to create analytics, serverless, and application integration solutions on AWS quickly.
Use cases
Ingest and collect terabytes of data per day from application and service logs, clickstream data, sensor data, and in-app user events to power live dashboards, generate metrics, and deliver data into data lakes.
Build applications for high-frequency event data such as clickstream data, and gain access to insights in seconds, not days, using AWS Lambda or Amazon Managed Service for Apache Flink. Deliver streaming data directly to Apache Iceberg tables and query up-to-the-minute insights from your data lake without managing pipelines or compaction jobs.
Fan out the same events to consumers for independent event processing such as fulfilling orders, running inference on individual events, enriching records with downstream APIs, and updating search indexes. Give each application its own queue with acknowledgment, automatic redelivery, built-in dead-letter handling, and queue-level CloudWatch metrics. Add consumer instances as demand changes without managing shards, leases, or checkpoints, while stream consumers and managed data delivery continue using the same records.
Feed fresh Kinesis data into AI/ML pipelines with minute-level freshness and optimized Parquet files and Iceberg tables for cost-efficient model training and inference.