Databases

💼 Business Context

• Many core systems (inventory, pricing, legacy CRM) only exist as a database — there's no API or event stream to tap
• Change-data-capture keeps CXOS's copy fresh without touching the source system's performance
• Owned by Data Engineering, in partnership with whichever team owns the source database

🔌 Technical Overview

For databases that support it, a .NET Core connector uses native change-data-capture (SQL Server CDC, PostgreSQL logical replication) to stream row-level changes with minimal source-system load. Where CDC isn't available, a scheduled .NET Core extract job (Azure Functions Timer, using Dapper) pulls incremental changes via a watermark column. Both paths normalize through Cxos.Ingestion.Client.

💾 Sample Change Event

{
  "event": "row_updated",
  "event_id": "b9c4d2e1-3a7f-4b8c-9d1e-5c2a8f0b6e91",
  "timestamp": "2026-08-01T09:00:12Z",
  "context": { "channel": "database", "source": "inventory_db", "table": "product_stock" },
  "properties": { "sku": "sku-1029", "warehouse": "PUN-01", "quantity_on_hand": 42 }
}

🔗 Integration Points

• Native CDC (SQL Server Change Data Capture, PostgreSQL logical replication) where supported
• Scheduled watermark-based extract (.NET Core + Dapper, Azure Functions Timer) as a fallback
• Cxos.Ingestion.Client NuGet package
• Azure Event Hubs — change-event stream consumed by the Stream Worker

🧰 Services Consumed

• Owning microservice — Cxos.Connectors.Cdc (see the Full Application Service Map)
• No dedicated database — stateless connector (see Platform Connectors above)

⚠️ Non-Functional Considerations

• Scale: CDC is designed to add minimal read load to the source database; extract jobs are rate-limited to protect OLTP performance
• Latency: CDC-based sync runs in near-real-time (seconds to low minutes); watermark extracts typically run every 15-60 minutes
• Reliability: extract jobs track the last-processed watermark to resume safely; CDC connectors track replication log position
• Security/Privacy: database credentials are stored in Azure Key Vault with least-privilege read-only accounts; sensitive columns are excluded at configuration time

🎯 Enterprise Example

A retailer's legacy inventory database has no API. A CDC connector streams stock-level changes into CXOS in near-real-time, letting the Activation API suppress "back in stock" email triggers for SKUs that CDC shows were restocked and immediately sold out again — avoiding a wave of complaints from a stale batch-only inventory feed.