Azure Fabric :: Part 2 — Petabyte-Scale Ingestion

Petabyte-Scale E-commerce Ingestion with Fabric and Databricks

Part 2: Backfill, CDC, APIs and Streams

Controlled ingestion combines checkpoints, watermarks, replay and reconciliation before downstream use.

Part 1 set the architecture and planning baseline: approximately 1–3 PB of historical e-commerce data, 5–20 TB/day of changes and 15–25 source systems as an illustrative sizing scenario. This article turns that baseline into an ingestion strategy for orders, customers, inventory, payments, clickstream, partner APIs, JSON feeds and files. The focus is not a single giant copy job; it is repeatable migration waves that can resume, reconcile and feed the downstream control dashboard.

1. Choose the right movement pattern for each source

Source/workload	Starting pattern	Use it when	Watch for
Large relational tables	Fabric Data Factory Copy Job or pipeline Copy activity; partitioned full load followed by CDC or incremental watermark	You need a controlled copy into OneLake and the connector supports the source pattern.	Source IOPS, snapshot consistency, CDC log retention, table skew and destination write behavior.
Supported operational database needing near-real-time analytical access	Fabric Mirroring	The source and region are supported and managed replication fits the requirement.	Per-source prerequisites, table limitations, latency, security model and mirrored storage/operations.
Bulk files, JSON, Parquet, images and PDFs	Copy Job schema-agnostic/binary file copy into a raw landing zone; parse separately	You need durable, replayable file landing while preserving the original bytes.	File count, tiny-file overhead, overwritten files, checksums and folder enumeration time.
REST/SaaS APIs	Data Factory pipeline Copy activity with REST connector and pagination	The provider exposes a supported endpoint and scheduled or incremental extraction is sufficient.	Rate limits, page/cursor expiry, Retry-After, schema changes, token refresh and API backfill cost.
Clickstream and operational events	Event Hubs → Fabric Eventstream → Lakehouse/Eventhouse; Databricks streaming where its stateful processing is needed	You need continuous ingestion, event-time processing or low-latency operations.	Partition-key skew, checkpointing, duplicates, late events, retention and consumer lag.
Already-open data in another lake	OneLake shortcut	Referencing supported open-format data in place meets access, latency and ownership requirements.	Source availability, network locality, permissions, external throttling and query-time performance.

Use Copy Job for straightforward data movement, including supported incremental patterns. Use pipelines when movement must be orchestrated with branches, dependencies, validation or notifications. Use Eventstream for continuous events. Shortcuts and Mirroring can expose supported data without building a full custom copy path, but they are not substitutes for every transformation or migration requirement. CDC availability and preview status depend on the connector and tenant, so verify current support before committing to the design.