Data & Analytics

Data Freshness / Latency

Director Analytics CTO VP Engineering COO

Data Freshness (also called data pipeline latency) measures the elapsed time between when data is generated in a source system and when it is available for analysis in the data warehouse or reporting layer. Stale data causes analysts and business leaders to make decisions on outdated information, which is particularly damaging for time-sensitive operational decisions. Fresher data enables more responsive decision-making.

Different data domains require different freshness targets: marketing attribution data may need hourly updates, while financial reporting data may be acceptable at a daily batch.

Formula

Average Time from Data Source Event to Availability in Analytics Layer

Where It Lives

dbtData transformation pipeline with freshness checks and alerts
Fivetran / AirbyteConnector sync frequency and data lag monitoring
Monte CarloData observability including freshness anomaly detection
SnowflakeData warehouse with pipeline metadata for freshness tracking

What Drives It

ELT pipeline sync frequency (batch vs. streaming)
Source system API rate limits and extraction latency
Data transformation complexity and compute time
Pipeline failure rates and recovery time
Data volume growth requiring longer processing windows

Causal Analysis: Migrating from batch to streaming data pipelines causally reduces data latency and can be measured directly by comparing freshness before and after migration.

Benchmark

Real-time streaming pipelines target sub-minute freshness; operational reporting targets under 4 hours; strategic reporting is often acceptable at 24-hour batch intervals.

Common Mistake

Setting a single freshness SLA for all data domains without recognizing that different use cases have fundamentally different freshness requirements.

How different roles think about this metric

Each function reads Data Freshness / Latency through a different lens and takes different actions when it changes.

Director Analytics

The Director of Analytics sets freshness SLAs by domain and monitors pipeline health to ensure business users receive data within the defined windows.

CTO

The CTO makes architectural decisions about streaming vs. batch pipelines based on the business's freshness requirements and the cost-freshness trade-off.

VP Engineering

VP Engineering builds and maintains the data infrastructure that determines freshness and owns pipeline reliability as a component of overall data product SLA.

COO

The COO relies on fresh operational data for real-time decision-making and escalates freshness SLA breaches that affect operational reporting.

Common Questions About Data Freshness / Latency

Click any question to expand the answer.

What is the difference between batch and streaming data pipelines?

Batch pipelines collect and process data at scheduled intervals (hourly, daily), producing data that is at minimum that interval old. Streaming pipelines process data continuously as it is generated, enabling near-real-time analytics with sub-minute latency. Streaming is technically more complex and expensive; batch is simpler and sufficient for most strategic reporting needs. Most organizations use a combination based on data domain freshness requirements.

What is data pipeline observability?

Data pipeline observability is the ability to monitor the health, freshness, and accuracy of data pipelines and detect anomalies before they affect downstream analytics consumers. Observability tools like Monte Carlo and Great Expectations track freshness (is data arriving on schedule?), volume (is the expected row count present?), and schema (have column definitions changed?). Without observability, stale or broken data can go unnoticed until analysts discover discrepancies.

How should I prioritize which pipelines to make fresher?

Prioritize based on the business impact of stale data in each domain. Operational metrics used for real-time decisions (customer support queue, inventory levels, live campaign bidding) need the freshest data. Strategic and financial reporting used for weekly or monthly reviews can tolerate daily batch latency. Interview your data consumers to understand which decisions they are making and what data freshness they actually require.

What is the cost trade-off of streaming vs. batch processing?

Streaming pipelines are typically 3–10× more expensive to build and operate than equivalent batch pipelines because they require always-on infrastructure (Kafka, Flink, Spark Streaming), more complex state management, and higher engineering expertise. Before investing in streaming, validate that the business actually needs sub-hour data freshness. Many organizations discover that their "real-time" reporting needs can be met by hourly batch pipelines at a fraction of the cost.

Related Metrics

Metrics that are commonly analyzed alongside Data Freshness / Latency.

Dashboard Adoption Rate | Dashboard Adoption Rate DQS | Data Quality Score Attribution Model Coverage | Attribution Model Coverage Error Rate | Error Rate

Role guides that include this metric

See how each role uses Data Freshness / Latency in context with the full set of metrics they own.

Director Analytics Guide → CTO Guide → VP Engineering Guide → COO Guide →

get started

See What’s Actually Moving Your Data Freshness / Latency

askotter connects your data sources and applies causal analysis to tell you exactly why your metrics are changing, not just that they changed.

Book a demo