Batch Upload (S3, SFTP, API)

💼 Business Context

• Some data simply arrives in bulk, on a schedule, from systems with no real-time integration option
• Enables efficient historical backfill when onboarding a new source, rather than replaying millions of individual events
• Owned by Data Engineering

🔌 Technical Overview

Batch Upload accepts files via Azure Blob Storage (direct upload or S3-compatible API), SFTP (for partners who require it, backed by Azure Blob Storage's native SFTP support), or a bulk API pull triggered on a schedule. A Blob-triggered .NET Core Azure Function validates and streams the file's records through Cxos.Ingestion.Client in batches — the same underlying ingestion contract as every real-time transport, just delivered on a schedule instead of continuously.

💾 Batch Job Summary

{
  "batch_id": "batch_20260801_0300",
  "path": "sftp",
  "file": "partner_export_2026-08-01.csv",
  "rows_ingested": 184213,
  "rows_quarantined": 12,
  "duration_seconds": 94
}

🔗 Integration Points

• Azure Blob Storage — primary landing zone, S3-compatible API for partners already using that pattern
• SFTP endpoint — Azure Blob Storage's native SFTP support for partners requiring it
• Blob-triggered Azure Function (.NET Core) — validation and batch ingestion
• Cxos.Ingestion.Client NuGet package — batch mode, same contract as real-time transports

🧰 Services Consumed

• Owning microservice — Cxos.Ingestion.Api (see the Full Application Service Map)
• Database — Azure Cache for Redis (cache only)

⚠️ Non-Functional Considerations

• Scale: large files are streamed and chunked rather than loaded into memory, supporting multi-million-row batches
• Latency: inherently batch — processed on file arrival or schedule, not real-time
• Reliability: partial-batch failures quarantine only the affected rows, not the whole file
• Security/Privacy: SFTP/Blob credentials are scoped per-partner and rotated on a schedule via Azure Key Vault

🎯 Enterprise Example

A newly onboarded partner delivers three years of historical transaction data as a single SFTP file drop. Batch Upload validates and ingests 8M+ rows overnight, giving the Identity API a complete historical view before the partner's real-time integration even goes live.