Private Kafka to Microsoft Fabric: what VNet injection actually costs you

Connecting Microsoft Fabric Real-Time Intelligence to a Kafka broker that lives on a private network — with no public endpoint anywhere in the path — sounds like a checkbox exercise. It is not. Eventstream VNet injection has a set of sharp edges that each cost hours, and several of them contradict the documentation outright.

fabric-realtime is a working reference lab for exactly this: the infrastructure is Bicep, the Fabric side is a documented portal walkthrough (because Fabric does not expose the necessary objects through ARM or a usable REST surface), and every claim in it was verified against a live build rather than inferred from docs.

What the lab builds

• A private Kafka broker behind an internal CA, with both a SASL_SSL listener and an mTLS listener, so you can test either authentication path.
• A subnet correctly delegated to the Eventstream streaming connector.
• Key Vault with a private endpoint, holding the CA and client certificates where Eventstream expects to find them.
• A layered network verification script that tells you which layer is broken (DNS → TCP → TLS → SASL), rather than leaving you to guess.
• A realistic event generator, so the resulting report shows signal instead of uniform noise.

It deploys into a spoke the platform team owns, not one it creates — the VNet, subnets, delegation, NSGs and Virtual WAN connection are vended against a spoke spec and consumed by resource ID, never modified. That is the realistic enterprise shape, and it constrains the design in ways a greenfield lab never surfaces.

The findings that change designs

The Kafka source has exactly one authentication kind: API Key

Eventstream’s Apache Kafka source offers no OAUTHBEARER, no Entra ID, no managed identity, no workspace identity for authenticating to the broker. Just a Key and a Secret over SASL_SSL or SSL (mTLS). Internal CA and mTLS are supported, which is good news for a real enterprise Kafka estate. But point that same source at an Azure Event Hubs Kafka endpoint and you are forced onto SAS keys — which a hardened tenant can quietly forbid.

A governed tenant can silently pin disableLocalAuth = true

Observed in a centrally governed subscription: an Event Hubs namespace created with disableLocalAuth = true by default, with no visible policy assignment at any scope you can see. Three separate code paths — the CLI namespace update, az resource update, and a raw ARM PATCH — each reported success and changed nothing.

Do not write a template that declares disableLocalAuth: false in a tenant like this. It produces a template that lies about the state it achieves, and a deployment that reports success while leaving the connector unusable.

Selecting a streaming gateway REMOVES the workspace identity auth option

This one contradicts the documentation directly. The docs say the Azure Event Hubs source, at Extended feature level, offers workspace identity for both public and private cases. In practice:

Choosing the streaming virtual network data gateway removes workspace identity from the list. The two features are documented as composable and are not. In a governed tenant that mandates both private endpoints and disabled local auth — a completely ordinary pairing — Azure Event Hubs cannot currently be used as an Eventstream source at all. That is the finding that pushes the design onto a Kafka broker you control, which is why the lab builds a real one.

Verify the network before blaming Fabric

Because there is no test-connection button when a data gateway is involved, the failure surface is opaque. The lab’s layered check isolates the actual fault:

A hostname that resolves to a public IP means private DNS is not in play, regardless of what the portal shows. Knowing precisely which layer broke is the difference between a five-minute fix and an afternoon.

Smaller edges, each an hour

• The delegation name is Microsoft.MessagingConnectors/connectors, not the portal label “Messaging Connectors.” Transcribe the docs literally and you get a rejection.
• The resource provider advertises itself as global, so regional availability is invisible unless you query list-available-delegations directly.
• There are two “VNet data gateways” — the Power Platform one and the streaming one — and only the streaming one is correct for Eventstream.
• The VNet and subnet are immutable the moment the Eventstream is published. Size the subnet from the partition count, not from habit.
• kafka-storage.sh --standalone does not exist before Kafka 4.0 — on 3.8.x it fails after all the certificate work, so the failure reads as a PKI problem when it is not.

Proven end to end

The whole point of the lab, demonstrated with evidence from both ends rather than a green tick in a portal: an Eventstream created entirely through the Fabric REST API, a service association link appearing on the delegated subnet (provisioningState: Succeeded), and events flowing from a private Kafka broker into Fabric over a path with no public endpoint. The full set of fifteen findings — each observed, not assumed — lives in docs/findings.md.

Get the repo

Everything is on GitHub, Bicep and scripts included, ready to point at your own vended spoke:

→ github.com/quintindk/fabric-realtime