A modern dilemma: Consume data from a stream or a lake

Choose the right data consumption pattern because rearchitecting is expensive.

Data is a critical component of software architecture, providing the foundation for business processes, including those powered by artificial intelligence (AI). Quality, volume, latency, consistency, ordering and duplicates are essential considerations in software design. Modern cloud architectures consume data in various ways, such as through application programming interfaces (APIs), streams and data lakes. You can consume data at the individual event or record level as it arrives or in larger batches. Legacy architectures often process data in scheduled batches, forcing surrounding architecture and business processes to depend on that approach. Getting the data consumption approach right up front is vital, because rearchitecting later is costly.

General patterns to consume data

There are several general data consumption patterns. These include APIs (record-by-record or multirecord results), event stream consumption (record-by-record), Delta Change Data Feed (CDF) consumption (record-by-record) or traditional batch ETL—extract, transform, load (multirecord results).

  • APIs are ideal for low-latency, record-by-record requirements. They also support multirecord results, typically through pagination. APIs usually follow a synchronous approach, where the client makes a call and waits for a response before moving to the next step. Model Context Protocol is an industry specification for agents to consume data and can support record-by-record or batch results via pagination, similar to APIs. They are often a wrapper on top of APIs.

  • The event stream pattern works well for sub-second latency requirements and typically runs on a tech stack like Kafka, Flink or Spark Streaming. These technologies excel at record-level calculations (like on-the-fly calculation), transformations (like column mapping) or enrichment (like adding data to an existing event).

  • Delta Lake is often used in data lakes and serves as a transaction log to standard Apache Parquet data files (source). CDF is a Delta Lake feature that tracks row-level changes (inserts, updates and deletes) between table versions (source). Delta CDF consumption is an alternative for handling record-level calculations compared to the event stream pattern, with a few key differences. Latency can still be low but rarely sub-second. The tech stack consists only of a Spark job and an Amazon S3 bucket, simplifying complexity and maintenance. This pattern provides higher cost efficiency for high-volume data. At extreme volumes (2 million per second), streaming can delay, especially during event replay.

  • Traditional batch ETL handles record groups and performs joins across many source datasets. Similar to Delta CDF, Spark and S3 can implement batch ETL. The key difference is the presence of many more source and destination datasets. These jobs run at massive scale with significant compute power.

Consistency, ordering and duplicates

Many considerations inform data consumption design. Consistency is one of the first to be defined: Does your design need strong or eventual consistency? Strong consistency means the process requires a response before proceeding to get the current state as of that exact moment and is often blocking or synchronous. Eventual consistency means processing will eventually get a response and is asynchronous (nonblocking). See the illustration below for details.

A side-by-side diagram comparing Strong Consistency and Eventual Consistency data models. The Strong Consistency section shows a client update synchronizing across all nodes before returning a successful response to the client. The Eventual Consistency section shows the primary node returning an immediate acknowledgment to the client, followed by asynchronous propagation to other nodes over time.

Image: Strong consistency vs. eventual consistency

Ordering

Order is another important consideration. Do you need to consume data in the exact order it was produced, or does order not matter? It is essential to understand which technology patterns preserve the original order of the data so consumers can take appropriate action. Kafka, for example, does not guarantee order across partitions when using multiple partitions.

A side-by-side diagram comparing Single Partition and Multiple Partitions message processing. The Single Partition side illustrates messages being written and read in a strict, guaranteed sequence. The Multiple Partitions side shows messages divided across two separate partitions; while order is preserved within each individual partition, the consumer reads them in an interleaved manner, demonstrating that global order is not guaranteed.

Image: Kafka ordering within a single partition vs. across multiple partitions

Duplicates

At-least-once delivery and exactly-once delivery describe the nature of duplicates. At-least-once delivery guarantees data delivery at least once, meaning it could arrive multiple times and result in duplicates. Exactly-once delivery guarantees the message will be delivered only once. Kafka Streams supports exactly-once delivery. Consumers need these details to set proper expectations and design processes appropriately. With at-least-once delivery patterns, producers can use data keys that enable consumers to deduplicate data. With data lakes, one approach to eliminating duplicate records is an upsert mode that checks keys to see whether a record already exists. If it does, the system updates it; if not, it inserts it.

A side-by-side diagram comparing At-Least-Once Delivery and Exactly-Once Delivery. The At-Least-Once Delivery side shows a failed acknowledgment prompting a producer to resend a message, resulting in duplicate messages at the consumer. The Exactly-Once Delivery side shows multiple identical messages being sent, but a de-duplication and transactional state filter ensures the consumer processes exactly one instance of the message.

Image: At-Least-Once vs. Exactly-Once Delivery comparison

Use case characteristics of consumption patterns

We categorize our five general patterns into two buckets: traditional batch ETL processing and real-time processing (e.g., APIs, event streams, Delta CDF). The use case characteristics below inform when data should be consumed via batch or a real-time pattern.

Traditional batch ETL processing

Batch ETL processing executes against a large batch set of data rather than individual records as they arrive. The following use case characteristics are a good fit:

  • Processing requires more than a day’s worth of historical data. This data is used within transformation logic, enabling sophisticated calculations, trend analysis and the derivation of insightful features based on historical patterns and context. Doing this in a stream takes longer and may not scale, depending on data volume.

  • Processing requires an entire day’s worth of data. These use cases require collecting an entire dataset before processing to ensure data quality, consistency and adherence to downstream formats.

  • Joining large datasets spanning large time windows. This performs efficient, scalable joins across extensive datasets, using optimized algorithms and distributed computing frameworks to handle complex relationships without compromising performance.

Real-time processing (e.g., APIs, event stream, Delta CDF)

Real-time processing requires immediate data handling and transformation as it arrives, enabling timely insights and actions. The following use case characteristics are a good fit:

  • Executing row-level transformations. This involves applying specific functions or logic to individual data records upon ingestion. Transformations can include data cleansing, standardization, enrichment or the creation of derived fields. This granular processing ensures data accuracy and prepares it for subsequent analysis or integration.

  • Performing data transformations upon API calls. This allows dynamic data manipulation based on external information retrieved through API interactions. As real-time data arrives, the system makes a call to an external service to fetch additional context or perform specific processing steps. These API results further transform incoming data, enabling richer processing.

  • Initiating real-time API calls based on incoming data streams. This feature enables the immediate application of logic or models when receiving incoming data streams. By invoking inference APIs in real time, organizations generate predictions, classifications or anomaly detections as events occur. This is vital for proactive decision-making, personalized experiences and automated responses.

  • Executing real-time joins between a data stream and a static dataset. This join combines a continuously flowing stream of real-time data with a relatively unchanging dataset. This enriches streaming data with reference information, such as looking up customer details, product information or geographical location. The static nature of one dataset allows efficient, rapid lookups during real-time processing.

Example use cases using the guidance above

Let’s walk through several use case situations and describe how to solve them.

Example 1. Duplicates situation: An application could introduce duplicates on a stream during failover situations. 
Recommendation: There are several ways to address this. The first requires consumers to process the data idempotently, meaning the business process can handle processing the same data twice. When this isn’t possible, consumers may need a deduplication mechanism using keys from the data.

Example 2. Data integrity situation: Ensure the data consumed by an application matches what the data producers provided. 
Recommendation: A traditional approach for data files involves the producer calculating a checksum of the contents included in the file header. The consumer recalculates that checksum and verifies that it matches the producer’s value. When consuming from a data stream, you do this on a narrower scope—on a record-by-record basis or over a window of data.

Example 3. Joining data across streams situation: Data across multiple streams needs to be joined and enriched.
Recommendation: Flink is a common technology that handles this exact use case. It provides window joins, which join the elements of two streams that share a common key and are in the same window (source). Flink uses a window assigner to define the window, and elements from both datasets pass to a user-defined <JoinFunction> where the user accesses results meeting the join criteria (source). Here is an example in Flink:

stream.join(otherStream) .where(<KeySelector>) .equalTo(<KeySelector>) .window(<WindowAssigner>) .apply(<JoinFunction>);

If data records arrive outside the specific window, there are a few options. Flink can keep the window state alive past the watermark using .allowedLateness(Time). It can also route those records to a side output using .sideOutputLateData. If this is a key requirement, it may be a reason to use a batching approach rather than consuming from a stream.

Example 4. Near real-time or micro-batching situation: Use cases need a pattern between real-time and batch processing.
Recommendation: It is common to have use cases where real-time and traditional batch processing are not a fit. In those scenarios, a hybrid pattern often works best. Micro-batching is a valid alternative that includes characteristics of both real-time and batch processing. In this example, records are batched and transmitted at a certain frequency (e.g., every 5 minutes). This optimizes the process by providing data in near real time when a threshold is met rather than waiting for once-a-day processing.

A graphic comparing “traditional batch processing,” where many tasks are grouped into “1 large batch,” with “micro batch processing,” where the same tasks are divided into three smaller “1 micro batch” groups.

Image: Micro-batch processing

Important requirements for data producers

Data producers can implement requirements that make various data consumption patterns viable for consumers. Operationally, it is critical to have appropriate monitoring and service-level agreements (SLAs). Data quality thresholds should be monitored and acted upon. SLAs are necessary for data delivery timeliness and error resolution. Data producers should ensure 100% accuracy and completeness of published data and agree with the consumer on how to handle duplicates. Data attributes should be agreed upon with the consumer around unique IDs, enabling consumers to identify duplicates if needed.

Conclusion

When deciding whether to use batch or real-time patterns, understand your use case characteristics to align them with the right approach. Consistency, ordering and duplicates are critical details in this decision. Designing the data consumption architecture correctly from the start avoids costly rework later. I hope this guidance helps with your use cases.

Acknowledgments

Special thanks to my colleagues who contributed to this blog post: Eric Helvey, Rahul Bobbasani, Sankarraj Lakshmanan and Jim Lebonitte.


Andrew Bonham
Andrew Bonham, Sr. Staff Full-stack Engineer

Andy Bonham is a senior staff full-stack engineer at Capital One with over 25 years of technology experience. His expertise includes cloud native architectures, platforms, open source BPM, real-time decisioning and rules engines. He holds both an undergraduate and a master’s degree in computer science with a concentration in information security as well as several industry certifications, including AWS Certified Solutions Architect – Associate, CISSP and CSSLP. He recently spoke at All Things Open 2026 and has written several technical articles for Capital One Tech.

Related Content

Open source tree diagram against light green background
Article | June 24, 2026 |7 min read
An abstract circular starburst graphic made of white radiating lines on a dark red background.
Article | October 9, 2026 |5 min read
Abstract graphic on a blue background featuring vertical white lines, audio-equalizer-style bars and six dark blue dots arranged in an arch formation.
Article | September 17, 2026 |10 min read