Ingestion Layer → Connectors

Custom Connectors

The framework for building a new source or destination integration when no pre-built connector exists, using the same contracts and tooling.

High-Level Design

Custom connectors extend the platform without leaving its standard patterns.

Data Source
Any External System
Proprietary or long-tail integration
→
Ingestion
Custom Connector Framework
IConnector interface + scaffolding
→
Processing
.NET Core Ingestion API
Receives the connector's normalized events
→
Foundation
Data Lakehouse
Source data alongside every other channel
→
Intelligence
.NET Core Analytics API
Blended with behavioral data
→
Activation
Reverse ETL (optional)
Where the integration requires it

💼 Business Context

  • Ensures a one-off or proprietary integration doesn't become a maintenance orphan outside the platform's standard patterns
  • Lets teams extend the platform to long-tail or internal systems without waiting on the core platform team's roadmap
  • Owned by whichever team needs the integration, with architectural review from Platform Engineering

🔌 Technical Overview

A custom connector is a .NET Core project scaffolded from an internal template implementing the same IConnector interface as every pre-built connector, referencing Cxos.Ingestion.Client directly for outbound events. The scaffolding includes the standard cross-cutting concerns (Key Vault credential resolution, Polly retry, structured logging, health-check endpoint) so a team only has to implement the source-specific extract/transform logic.

Scaffolding Includes

IConnector interface Key Vault credential resolution Polly retry policy Health-check endpoint

💾 Custom Connector Skeleton

public class AcmePartnerConnector : IConnector
{
    public async Task SyncAsync(CancellationToken ct)
    {
        var records = await _acmeClient.GetUpdatesAsync(ct);
        foreach (var r in records)
            await _cxosClient.TrackAsync(MapToEvent(r));
    }
}

🔗 Integration Points

  • IConnector interface + internal scaffolding template
  • Cxos.Ingestion.Client NuGet package
  • Azure Key Vault — credential storage, same pattern as pre-built connectors
  • Platform Engineering architectural review — before a custom connector goes to production

🧰 Services Consumed

  • Owning microservice — Cxos.Connectors.* (registry family) (see the Full Application Service Map)
  • No dedicated database — stateless connector (see Platform Connectors above)

⚠️ Non-Functional Considerations

  • Scale: no different from a pre-built connector once built — same deployment and scaling model (Azure Container Apps or AKS)
  • Latency: determined by the connector author's chosen sync mode (webhook vs. polling)
  • Reliability: required to implement the same health-check contract as pre-built connectors so it participates in standard platform monitoring
  • Security/Privacy: subject to the same architectural review as any new integration — credential handling and PII mapping are checked before go-live

🎯 Enterprise Example

A team needs to integrate a proprietary in-house loyalty system with no equivalent pre-built connector. Using the scaffolding, an engineer ships a working custom connector in two days instead of building ingestion plumbing from scratch — inheriting retry, credential management, and monitoring for free.

← Back to Connectors