A modern dilemma: Consume data from a stream or a lake
Choose the right data consumption pattern because rearchitecting is expensive.

Data is a critical component of software architecture, providing the foundation for business processes, including those powered by artificial intelligence (AI). Quality, volume, latency, consistency, ordering and duplicates are essential considerations in software design. Modern cloud architectures consume data in various ways, such as through application programming interfaces (APIs), streams and data lakes. You can consume data at the individual event or record level as it arrives or in larger batches. Legacy architectures often process data in scheduled batches, forcing surrounding architecture and business processes to depend on that approach. Getting the data consumption approach right up front is vital, because rearchitecting later is costly.
General patterns to consume data
There are several general data consumption patterns. These include APIs (record-by-record or multirecord results), event stream consumption (record-by-record), Delta Change Data Feed (CDF) consumption (record-by-record) or traditional batch ETL—extract, transform, load (multirecord results).
-
APIs are ideal for low-latency, record-by-record requirements. They also support multirecord results, typically through pagination. APIs usually follow a synchronous approach, where the client makes a call and waits for a response before moving to the next step. Model Context Protocol is an industry specification for agents to consume data and can support record-by-record or batch results via pagination, similar to APIs. They are often a wrapper on top of APIs.
-
The event stream pattern works well for sub-second latency requirements and typically runs on a tech stack like Kafka, Flink or Spark Streaming. These technologies excel at record-level calculations (like on-the-fly calculation), transformations (like column mapping) or enrichment (like adding data to an existing event).
-
Delta Lake is often used in data lakes and serves as a transaction log to standard Apache Parquet data files (source). CDF is a Delta Lake feature that tracks row-level changes (inserts, updates and deletes) between table versions (source). Delta CDF consumption is an alternative for handling record-level calculations compared to the event stream pattern, with a few key differences. Latency can still be low but rarely sub-second. The tech stack consists only of a Spark job and an Amazon S3 bucket, simplifying complexity and maintenance. This pattern provides higher cost efficiency for high-volume data. At extreme volumes (2 million per second), streaming can delay, especially during event replay.
-
Traditional batch ETL handles record groups and performs joins across many source datasets. Similar to Delta CDF, Spark and S3 can implement batch ETL. The key difference is the presence of many more source and destination datasets. These jobs run at massive scale with significant compute power.
Consistency, ordering and duplicates
Many considerations inform data consumption design. Consistency is one of the first to be defined: Does your design need strong or eventual consistency? Strong consistency means the process requires a response before proceeding to get the current state as of that exact moment and is often blocking or synchronous. Eventual consistency means processing will eventually get a response and is asynchronous (nonblocking). See the illustration below for details.
Image: Strong consistency vs. eventual consistency
Ordering
Order is another important consideration. Do you need to consume data in the exact order it was produced, or does order not matter? It is essential to understand which technology patterns preserve the original order of the data so consumers can take appropriate action. Kafka, for example, does not guarantee order across partitions when using multiple partitions.
Image: Kafka ordering within a single partition vs. across multiple partitions
Duplicates
At-least-once delivery and exactly-once delivery describe the nature of duplicates. At-least-once delivery guarantees data delivery at least once, meaning it could arrive multiple times and result in duplicates. Exactly-once delivery guarantees the message will be delivered only once. Kafka Streams supports exactly-once delivery. Consumers need these details to set proper expectations and design processes appropriately. With at-least-once delivery patterns, producers can use data keys that enable consumers to deduplicate data. With data lakes, one approach to eliminating duplicate records is an upsert mode that checks keys to see whether a record already exists. If it does, the system updates it; if not, it inserts it.
Image: At-Least-Once vs. Exactly-Once Delivery comparison
Use case characteristics of consumption patterns
We categorize our five general patterns into two buckets: traditional batch ETL processing and real-time processing (e.g., APIs, event streams, Delta CDF). The use case characteristics below inform when data should be consumed via batch or a real-time pattern.
Traditional batch ETL processing
Batch ETL processing executes against a large batch set of data rather than individual records as they arrive. The following use case characteristics are a good fit:
-
Processing requires more than a day’s worth of historical data. This data is used within transformation logic, enabling sophisticated calculations, trend analysis and the derivation of insightful features based on historical patterns and context. Doing this in a stream takes longer and may not scale, depending on data volume.
-
Processing requires an entire day’s worth of data. These use cases require collecting an entire dataset before processing to ensure data quality, consistency and adherence to downstream formats.
-
Joining large datasets spanning large time windows. This performs efficient, scalable joins across extensive datasets, using optimized algorithms and distributed computing frameworks to handle complex relationships without compromising performance.
Real-time processing (e.g., APIs, event stream, Delta CDF)
Real-time processing requires immediate data handling and transformation as it arrives, enabling timely insights and actions. The following use case characteristics are a good fit:
-
Executing row-level transformations. This involves applying specific functions or logic to individual data records upon ingestion. Transformations can include data cleansing, standardization, enrichment or the creation of derived fields. This granular processing ensures data accuracy and prepares it for subsequent analysis or integration.
-
Performing data transformations upon API calls. This allows dynamic data manipulation based on external information retrieved through API interactions. As real-time data arrives, the system makes a call to an external service to fetch additional context or perform specific processing steps. These API results further transform incoming data, enabling richer processing.
-
Initiating real-time API calls based on incoming data streams. This feature enables the immediate application of logic or models when receiving incoming data streams. By invoking inference APIs in real time, organizations generate predictions, classifications or anomaly detections as events occur. This is vital for proactive decision-making, personalized experiences and automated responses.
-
Executing real-time joins between a data stream and a static dataset. This join combines a continuously flowing stream of real-time data with a relatively unchanging dataset. This enriches streaming data with reference information, such as looking up customer details, product information or geographical location. The static nature of one dataset allows efficient, rapid lookups during real-time processing.
Example use cases using the guidance above
Let’s walk through several use case situations and describe how to solve them.
Example 1. Duplicates situation: An application could introduce duplicates on a stream during failover situations.
Recommendation: There are several ways to address this. The first requires consumers to process the data idempotently, meaning the business process can handle processing the same data twice. When this isn’t possible, consumers may need a deduplication mechanism using keys from the data.
Example 2. Data integrity situation: Ensure the data consumed by an application matches what the data producers provided.
Recommendation: A traditional approach for data files involves the producer calculating a checksum of the contents included in the file header. The consumer recalculates that checksum and verifies that it matches the producer’s value. When consuming from a data stream, you do this on a narrower scope—on a record-by-record basis or over a window of data.
Example 3. Joining data across streams situation: Data across multiple streams needs to be joined and enriched.
Recommendation: Flink is a common technology that handles this exact use case. It provides window joins, which join the elements of two streams that share a common key and are in the same window (source). Flink uses a window assigner to define the window, and elements from both datasets pass to a user-defined <JoinFunction> where the user accesses results meeting the join criteria (source). Here is an example in Flink:
stream.join(otherStream) .where(<KeySelector>) .equalTo(<KeySelector>) .window(<WindowAssigner>) .apply(<JoinFunction>);
If data records arrive outside the specific window, there are a few options. Flink can keep the window state alive past the watermark using .allowedLateness(Time). It can also route those records to a side output using .sideOutputLateData. If this is a key requirement, it may be a reason to use a batching approach rather than consuming from a stream.
Example 4. Near real-time or micro-batching situation: Use cases need a pattern between real-time and batch processing.
Recommendation: It is common to have use cases where real-time and traditional batch processing are not a fit. In those scenarios, a hybrid pattern often works best. Micro-batching is a valid alternative that includes characteristics of both real-time and batch processing. In this example, records are batched and transmitted at a certain frequency (e.g., every 5 minutes). This optimizes the process by providing data in near real time when a threshold is met rather than waiting for once-a-day processing.
Image: Micro-batch processing
Important requirements for data producers
Data producers can implement requirements that make various data consumption patterns viable for consumers. Operationally, it is critical to have appropriate monitoring and service-level agreements (SLAs). Data quality thresholds should be monitored and acted upon. SLAs are necessary for data delivery timeliness and error resolution. Data producers should ensure 100% accuracy and completeness of published data and agree with the consumer on how to handle duplicates. Data attributes should be agreed upon with the consumer around unique IDs, enabling consumers to identify duplicates if needed.
Conclusion
When deciding whether to use batch or real-time patterns, understand your use case characteristics to align them with the right approach. Consistency, ordering and duplicates are critical details in this decision. Designing the data consumption architecture correctly from the start avoids costly rework later. I hope this guidance helps with your use cases.
Acknowledgments
Special thanks to my colleagues who contributed to this blog post: Eric Helvey, Rahul Bobbasani, Sankarraj Lakshmanan and Jim Lebonitte.



