Snowflake / BigQuery

💼 Business Context

• Teams that already standardize on Snowflake or BigQuery for analysis get CXOS data without learning a new query engine
• Because the lakehouse is open Iceberg, Snowflake delivery can be a zero-copy table attachment rather than a duplicated, staling export — the same zero-copy principle from the Unified Data Foundation extended to an external consumer
• Owned by Data Engineering / Analytics Engineering

🔌 Technical Overview

For Snowflake, delivery is a governance-reviewed grant of Iceberg external-table access directly against the lakehouse's Azure Data Lake Storage Gen2 location — no data movement at all, consistent with the platform's zero-copy architecture. BigQuery, lacking native Iceberg external-table support at the same maturity, instead receives a scheduled load job (a .NET Core Docker container on Azure Container Apps Jobs) that reads via the Query & Analytics Engine and writes into a BigQuery-native table on a schedule.

💾 Warehouse Delivery Config

-- Snowflake: zero-copy external table
CREATE EXTERNAL TABLE partner_db.order_fact
  LOCATION = 'abfss://lakehouse@cxosdata.dfs.core.windows.net/marts/order_fact'
  FILE_FORMAT = ICEBERG;

-- BigQuery: scheduled load job (no native Iceberg external tables)
{ "destination": "bigquery://partner-project.cxos.order_fact", "schedule": "hourly" }

🔗 Integration Points

• Open Table Format (Iceberg) — what makes the Snowflake zero-copy path possible
• Query & Analytics Engine — read path for the BigQuery load job
• Governance & Security — reviews and grants external-table access before any warehouse can attach
• Azure Data Lake Storage Gen2 — the physical location Snowflake attaches to directly

🧰 Services Consumed

• Owning microservice — Cxos.Connectors.BatchExport (see the Full Application Service Map)
• No dedicated database — stateless connector (see Platform Connectors above)

⚠️ Non-Functional Considerations

• Scale: Snowflake attachment has no export-size limit since no copy is made; BigQuery load jobs are sized to the scheduled table/range like any batch export
• Latency: Snowflake external tables reflect the lakehouse in near-real-time (as fast as the underlying dbt refresh); BigQuery lags by its load schedule
• Reliability: Snowflake's zero-copy path has no separate copy to drift out of sync with the source; the BigQuery path is validated like any other batch export
• Security/Privacy: external-table grants are entitlement-scoped and reviewed by Governance & Security before being issued, same as any other access grant

🎯 Enterprise Example

A partner analytics team standardized on Snowflake asks for order data access. Rather than building a recurring export pipeline, Data Engineering grants a reviewed Iceberg external-table connection — the partner queries current data directly, with zero ongoing export maintenance.