Data quality is often overlooked. Teams devote significant time to developing a feature, creating pipelines, and setting up dashboards, only addressing the accuracy of the underlying data when a stakeholder identifies a questionable figure. By then, rectifying it becomes substantially more expensive.
This issue is widespread, affecting engineering teams of all sizes, and the repercussions can include wasted resources and lost trust from leadership in the data team. Most of these setbacks are avoidable if data quality is prioritized from the beginning rather than as a secondary task.
The typical process of data projects
Understanding common data engineering project initiation can clarify the issue. Typically, it starts with a cross-functional discussion about a new feature and desired metrics. The data team collaborates with data scientists and analysts to define key metrics. Engineers determine what is feasible to instrument and any constraints. A data engineer then drafts a logging specification detailing what events to capture, which fields to include, and their importance.
This logging specification acts as a reference for everyone. Downstream users depend on it. When effective, the whole system operates smoothly.
Before entering production, a validation phase occurs in development and staging environments. Engineers examine key interactions, ensure correct events with correct fields, fix issues, and repeat until everything is satisfactory. It’s laborious but designed to be a safety measure.
The issue arises afterward.
The disparity between staging and production
Once data is live and ETL pipelines are operational, most teams assume the data contract from instrumentation will persist. It seldom does, certainly not indefinitely.
A typical scenario involves expecting an event to fire when a user performs an action. Months later, a server-side change alters this timing, causing the event to fire earlier with different key field values. Nobody identifies it as a data-impacting change. The pipeline runs on, and dashboards continue to receive data.
Weeks or months later, someone notices flat metrics. A data scientist investigates, traces the issue, and identifies the root cause. The team now faces full remediation: updating ETL logic, backfilling impacted partitions, and explaining to stakeholders how long the numbers were incorrect.
The steep cost of a single missed change includes engineering analysis time, codebase updates, backfill compute resources, and more seriously, diminished trust in the data team. Once stakeholders encounter inaccurate numbers, they begin doubting everything. Regaining that confidence is challenging.
This pattern is common in large systems with independent microservices evolving on their own release schedules, causing a gradual shift between pipeline expectations and actual data content.
Why validation shouldn’t end at staging
The main problem is treating data validation as a singular step rather than a continuous process. Staging validation matters, but it only confirms the system state at one moment. Production is ever-changing.
Data quality needs enforcement at every pipeline layer, from data production through transport to fully processed tables consumers use. The current data tooling ecosystem supports making this feasible.
Ensuring quality at the source
The initial defense is a data contract at the production level. Enforcing a strict schema upon data emission, complete with typed fields and defined structure, catches breaking changes immediately rather than letting them silently propagate. Schema registries, commonly paired with streaming platforms like Apache Kafka, serialize data by a schema before transport and validate it upon deserialization. Compatibility checks maintain pipeline integrity during schema evolution.
Avro formatted schemas in a schema registry are widely supported for this reason, establishing an explicit, versioned contract between producers and consumers, enforced dynamically, not just noted in a spec file that could be ignored.
Write, audit, publish: A pipeline quality gate
In the processing stage, Apache Iceberg introduces a valuable data quality enforcement pattern known as Write-Audit-Publish, or WAP. Iceberg uses file metadata where every write is a tracked commit. WAP incorporates an audit step before data is deemed production-ready.
In practice, daily pipelines operate as follows. Raw data is collected in an ingestion layer, usually compiled from smaller time window partitions into a full daily partition. The ETL process retrieves this data, performs transformations like normalizations and timezone