Time Travel & Versioning

💼 Business Context

• Lets the business answer 'what did this customer's profile look like on this date' — critical for dispute resolution and compliance audits
• Makes backfills and schema changes safe by allowing validation against a snapshot before committing
• Owned by Data Engineering, used heavily by Compliance and Data Science

🔌 Technical Overview

Every write to an Iceberg table creates a new immutable snapshot rather than mutating existing data in place. Queries can target a specific snapshot ID or timestamp (SELECT * FROM table FOR TIMESTAMP AS OF ...), letting Backfills validate a new snapshot before switching the 'current' pointer, and letting Compliance reconstruct exactly what data looked like at any past moment.

💾 Time Travel Query

SELECT * FROM curated.customer_profile
FOR TIMESTAMP AS OF '2026-07-01T00:00:00Z'
WHERE customer_key = 'cust_004821';

🔗 Integration Points

• Apache Iceberg — snapshot mechanism underlying this capability
• Backfills — use snapshot isolation to validate before cutover
• Query & Analytics Engine — exposes the AS OF query syntax
• Audit Logs (Governance & Security) — often cross-referenced with a specific snapshot for investigations

🧰 Services Consumed

• Owning microservice — Cxos.Foundation.Infrastructure (see the Full Application Service Map)
• Database — Azure Data Lake Storage Gen2 (Iceberg) — the lakehouse itself

⚠️ Non-Functional Considerations

• Scale: snapshot metadata overhead is small relative to data size; old snapshots are expired on a retention policy to bound storage growth
• Latency: querying a historical snapshot has the same performance characteristics as querying current data
• Reliability: rollback to a prior snapshot is a metadata operation, not a data-copy operation — fast and safe
• Security/Privacy: historical snapshots are subject to the same access control and retention policy as current data — time travel doesn't bypass governance

🎯 Enterprise Example

A customer disputes a charge, claiming their subscription tier was different at the time. Compliance queries the customer profile table as of the disputed date and gets a definitive, auditable answer instead of relying on a change log that might be incomplete.