Design a data pipeline
Design a batch or streaming data pipeline from ingestion through transformation to a warehouse or reporting store. Handles late data, SLA freshness, and monitoring.
Preview
Copy prompt
Design a data pipeline that ingests data from [Source: event stream, database change feed, or REST API], applies [Describe the transformation logic], and loads the result into [Destination: data warehouse, data lake, or reporting database]. Decide between a batch and a streaming architecture based on the required data freshness SLA. Describe how the pipeline handles late-arriving or out-of-order events. Define the freshness SLA and the monitoring metrics and alerts that detect when the pipeline falls behind. Explain the operational runbook for recovering from a pipeline failure without losing or double-counting data.
More
Architecture
Prompts
