Lifecycle Management
Automated policies that age data through storage tiers and eventually expire it — balancing cost, performance, and retention requirements.
High-Level Design
Lifecycle Management is what makes retention policy actually enforced, not just documented.
💼 Business Context
- Keeps lakehouse storage costs proportional to actual value, not an ever-growing bill for data nobody queries anymore
- The mechanism that makes retention-limit and 'right to be forgotten' compliance requirements actually enforceable
- Owned by Data Engineering, policy set by Data Governance/Legal
🔌 Technical Overview
A scheduled .NET Core job applies lifecycle policies per table: moving older partitions from Azure Blob Storage's Hot tier to Cool/Archive tiers based on access-frequency assumptions, expiring old Iceberg snapshots beyond the time-travel retention window, and permanently deleting data that has exceeded its retention limit or is subject to an explicit deletion request (e.g., a GDPR erasure request), including rewriting any files that still contain the deleted records.
Policy Actions
💾 Lifecycle Policy
{
"table": "raw.events",
"hot_days": 30,
"cool_days": 180,
"archive_after_days": 365,
"delete_after_days": 2555
}
🔗 Integration Points
- Azure Blob Storage lifecycle policies — automated tiering
- Apache Iceberg snapshot expiration — bounds time-travel storage overhead
- Governance & Security — defines retention and erasure requirements
- Scheduled .NET Core job (Azure Functions Timer) — executes the policy
🧰 Services Consumed
- Owning microservice —
Cxos.Foundation.Infrastructure(see the Full Application Service Map) - Database — Azure Data Lake Storage Gen2 (Iceberg) — the lakehouse itself
⚠️ Non-Functional Considerations
- Scale: tiering and expiration run as background jobs sized to process the full lakehouse on a rolling schedule without impacting query performance
- Latency: not applicable — this is a background cost/compliance process, not a query-path concern
- Reliability: deletion requests are tracked and confirmed, since 'right to be forgotten' compliance requires proof of completion
- Security/Privacy: this is itself a core privacy control — retention limits and erasure requests are only meaningful if actually enforced
🎯 Enterprise Example
A customer submits a data deletion request under GDPR. The lifecycle management job locates every table containing their records, rewrites the affected files to exclude them, and the compliance team receives a confirmation report — turning a manual, error-prone process into an automated, auditable one.