Bryan van der Laan
CH.02 Writing9 min read#databricks#fabric#dbt#data-platforms

Before You Pick Fabric, Count Your Engineers

I work with both Microsoft Fabric and Databricks. That means real client projects, with different engineers, at the same time. So when someone asks me which one to pick, I don't have the luxury of a neutral answer.

This post isn't a verdict on Fabric as a whole. It's about one situation I keep running into: a small team of engineers, building with dbt, on a small Fabric capacity. That's where many of our clients start, and it's exactly where Fabric hurts the most. In that situation, I pick Databricks. This comes from issues I've been up against in recent months, and I'll say where I'm confident, where the sources back me up, and where I simply haven't seen Fabric's strengths up close.

Two developers, one capacity#

Spark compute in Fabric comes from your capacity. Each capacity unit maps to two Spark VCores, and every notebook, Spark job and lakehouse operation on that capacity draws from the same pool. An F64 gives you 128 VCores. An F8 gives you 16, and an F4 just 8.

On those small SKUs, the default starter pool is tiny: a single Medium node on an F4, two on an F8. That's too small for real transformation work, so we configure our own Spark pools. But a custom pool can't change the math. A session big enough for real work claims a large share of a 16-VCore pool before bursting, and the next developer gets what's left.

And it's a problem I keep running into. Not every client has the luxury of running an F32 right away, so we have to live with F4 or F8 capacities. On those, a second developer is often enough to trigger "too many requests" errors. That error is the capacity telling you it has no Spark cores left for another session.

It doesn't help that dbt claims its own share. The Fabric Lakehouse adapter runs dbt through the Livy API and keeps a Spark session open for the entire run. So every developer running dbt takes a session from the same small pool as everyone's notebooks, and the jobs that do get through slow to a crawl.

I've seen it happen more than once. Two developers, working on different features, both running dbt at the same time. Then the message comes in: "Are you using the capacity right now? I'm getting errors about usage." And then you wait. And wait some more. I could have built a house in the time I've spent waiting for capacity to free up.

Bursting helps for a while, but it's borrowed capacity. Fabric lets you pay back the overuse later, and if you keep borrowing, it starts delaying and eventually rejecting new interactive work. So your working day depends on what everyone else on the capacity happens to be doing.

The obvious fix is a bigger SKU. That works, but you're paying for capacity your data volume doesn't need, just so your team can work in parallel. You end up sizing for headcount, not for data.

Lakehouse or Warehouse: pick your trade-off#

Fabric gives you two places to build: the Lakehouse and the Warehouse. On paper that's flexibility. For a dbt team, both come with a catch.

The Lakehouse is Spark, so dbt runs through the Livy adapter, with all the capacity problems above. You do get a SQL analytics endpoint on top, but it's read-only. For dbt, that means every write goes through Spark. The endpoint also has to sync with the lakehouse, which usually takes less than a minute but can take minutes, and Microsoft's faster sync is still in preview.

The Warehouse is a T-SQL engine, so it avoids the Spark sessions. But it doesn't support every T-SQL statement you'd know from SQL Server, and as soon as you also need Spark, your team is building across two engines with two sets of rules.

Then there's Fabric's own answer for dbt: native dbt jobs, which run dbt directly inside Data Factory. It's a promising feature, and since August it can target the Lakehouse too. But dbt jobs are still in preview, and so is that Lakehouse support. I don't put client production on a preview feature.

On Databricks there's one engine, one catalog, and a dbt adapter that has been around for years. I don't have to pick between two halves of a platform before I've written a single model.

Always on, or waiting#

A Fabric capacity bills while it's running, whether anyone uses it or not. You can pause it at night and on weekends, but then everything on it is unavailable, reports included. Pausing isn't free either: all the usage Fabric had smoothed out over the coming hours gets added to your bill the moment you pause. And if you bought a reservation to get the discount, you pay during the pause anyway.

So you choose between paying around the clock, or building a schedule to turn your platform on and off. Even when it's running, the waiting doesn't fully stop. According to Microsoft, a session on a custom Spark pool takes about a minute to start. Custom live pools get that down to seconds by keeping clusters warm on a schedule, but they only support notebooks, and warm clusters are billed while they sit ready. When every warm cluster is busy, you're back to several minutes of provisioning.

Databricks has startup times too. A classic cluster needs a few minutes. The difference is what happens in between: an idle cluster shuts itself down and stops costing money, and serverless compute starts much faster. On Fabric, I'm paying for the capacity and still planning around the wait.

What about Autoscale Billing?#

Microsoft's answer to the contention problem is Autoscale Billing for Spark, generally available since mid-2025. Turn it on and Spark stops using your capacity. It runs on serverless compute instead, billed for the time jobs actually run, up to a maximum your capacity admin sets.

It does help with contention. It just isn't very predictable. That maximum limits how much Spark can run at once, not what you spend in a month, so the bill moves with however much your team happens to run. It comes on top of the capacity you still need for everything else, and it's billed at pay-as-you-go rates, so the roughly 41% discount of a reserved capacity doesn't apply. And once Spark hits the maximum, interactive work gets throttled again.

That flips the usual Fabric trade-off. Before, you had a predictable bill and unpredictable waiting. Now you get less waiting and a less predictable bill. At that point it's a consumption model, which is how Databricks has worked all along, with far more control over it.

How Databricks handles it#

On Databricks you get far more than one knob. You shape compute around the work:

  • Per developer. Each engineer gets their own cluster, sized for development work and set to shut itself down after a fixed amount of idle time.
  • Per job. Production runs get a job cluster that starts for that run and disappears when it's done, so a nightly load never competes with someone's afternoon of debugging.
  • Per use case. dbt and BI queries can share a SQL warehouse that adds clusters when queries start to queue and scales back down when it's quiet. Heavy transformations or ML get their own, bigger compute. And where you don't want to manage clusters at all, there's serverless.

Cluster policies tie it together: admins decide which sizes people can pick, enforce auto-termination and tag everything for cost tracking.

To be clear, Databricks also gets more expensive as your team grows. Every developer's cluster costs money, and idle clusters and cluster sprawl are real risks. But you pay for what each person actually runs, and nothing when nobody is working. Those are problems you can fix with configuration.

Working in parallel, part two#

Capacity isn't the only thing that gets painful as teams grow.

Fabric's Git integration links one workspace to one branch, so the natural setup is a personal feature workspace per developer. But as Xebia describes well, a fresh feature workspace pulls in your notebooks, pipelines and lakehouse definitions, without the data. Out of the box, every developer starts with an empty lakehouse.

There is a workable fix, and Xebia describes that too: keep the data in one central development lakehouse and point every feature workspace at it. It works, but it goes against what the Fabric UI nudges you toward, and it has sharp edges. Git sync can fail, for example, when a commit deletes an item that another workspace still references.

On Databricks, this is how it works out of the box. Developers point their own compute at the same governed tables in Unity Catalog, and keep their work separate in their own schemas or catalogs.

"But we need Power BI"#

I hear this one a lot, and honestly, it doesn't hold up anymore.

The native Power BI connector for Databricks has been generally available since 2021, with DirectQuery, Entra ID authentication and single sign-on. Setting it up takes a few clicks. I have found no difference in ease of use between Fabric and Databricks.

Fabric's biggest Power BI advantage is Direct Lake, which gives you import-level speed without scheduled refreshes. You can get it with Databricks too: Fabric can mirror your Azure Databricks Unity Catalog into OneLake without copying the data, and you build Direct Lake models on top of that. You'll still need a Fabric capacity and you'll have to set up permissions again on the Fabric side, but only for reporting. Your engineers stay on Databricks.

If Power BI is the only reason you're considering Fabric, it isn't a reason.

Is it worth migrating if you already chose Fabric?#

This is the harder question, because sunk cost is real and "the grass is greener" is not a strategy.

My test is simple: migrate if the gap is costing you something you need now, and the gap will still be there in two years.

For the teams I work with, the gap is concrete: waiting for capacity to free up, and not being able to work in parallel. Whether migrating is feasible depends mostly on where your logic lives. If your data is in Delta tables and your transformations are in dbt, moving platforms is mostly a matter of repointing compute and redoing the governance layer: service principals, secrets and permissions. It's a bit of setup, but it's bounded and you only do it once. Worth it every time, in my opinion, especially when your data platform or your team is still small.

It gets much harder if your logic lives in Fabric-native items: pipelines, dataflows, Power Query, notebooks with no dbt layer underneath. Then it's more about rewriting your platform than migrating it. If you're on Fabric today and not in that situation yet, keeping your transformations in dbt is the cheapest insurance you can buy, whatever you decide later.

What I haven't seen#

Beyond Direct Lake, I can't list Fabric's advantages from experience, because I keep spending my time on the walls. On a larger capacity, with a team that works in a single engine, a lot of these problems shrink. And a team that is Power BI first, with light transformation work and one person building, might genuinely be better off with Fabric's all-in-one simplicity. I just don't work with many of those teams.

Where I land#

For a consultancy, "which platform is better" is close to the wrong question. The client's existing investment and the team's skills matter more than any feature list, and sometimes that means Fabric.

But when I get to choose, for a small team of engineers building with dbt, I land on Databricks every time.

References#

Adegeest, R. (2025, December 5). Microsoft Fabric workspace setup for multi-developer teams. Xebia. https://xebia.com/blog/microsoft-fabric-workspace-setup-for-multi-developer-teams/

dbt Labs. (n.d.). Microsoft Fabric Lakehouse setup. dbt Developer Hub. Retrieved September 24, 2026, from https://docs.getdbt.com/reference/warehouse-profiles/fabricspark-profile

Kar_c. (n.d.). Concurrent Spark sessions [Online forum post]. Microsoft Fabric Community. https://community.fabric.microsoft.com/discussions/ac_dataengineering/concurrent-spark-sessions/4601239

Leone, S., Stuart, C., & Efeoglu, C. (2021, February 26). Announcing general availability (GA) of the Power BI connector for Databricks. Databricks Blog. https://www.databricks.com/blog/2021/02/26/announcing-general-availability-ga-of-the-power-bi-connector-for-databricks.html

Microsoft. (n.d.-a). Autoscale Billing for Spark in Microsoft Fabric. Microsoft Learn. Retrieved September 24, 2026, from https://learn.microsoft.com/en-us/fabric/data-engineering/autoscale-billing-for-spark-overview

Microsoft. (n.d.-b). Concurrency limits and queueing in Apache Spark for Fabric. Microsoft Learn. Retrieved September 24, 2026, from https://learn.microsoft.com/en-us/fabric/data-engineering/spark-job-concurrency-and-queueing

Microsoft. (n.d.-c). Configure starter pools in Microsoft Fabric. Microsoft Learn. Retrieved September 24, 2026, from https://learn.microsoft.com/en-us/fabric/data-engineering/configure-starter-pools

Microsoft. (n.d.-d). Custom live pools in Microsoft Fabric. Microsoft Learn. Retrieved September 24, 2026, from https://learn.microsoft.com/en-us/fabric/data-engineering/custom-live-pools-overview

Microsoft. (n.d.-e). dbt job in Microsoft Fabric (preview). Microsoft Learn. Retrieved September 24, 2026, from https://learn.microsoft.com/en-us/fabric/data-factory/dbt-job-overview

Microsoft. (n.d.-f). The Fabric throttling policy. Microsoft Learn. Retrieved September 24, 2026, from https://learn.microsoft.com/en-us/fabric/enterprise/throttling

Microsoft. (n.d.-g). Limitations of Fabric Data Warehouse. Microsoft Learn. Retrieved September 24, 2026, from https://learn.microsoft.com/en-us/fabric/data-warehouse/limitations

Microsoft. (n.d.-h). Microsoft Fabric pricing. Microsoft Azure. Retrieved September 24, 2026, from https://azure.microsoft.com/en-us/pricing/details/microsoft-fabric/

Microsoft. (n.d.-i). Mirroring Azure Databricks Unity Catalog. Microsoft Learn. Retrieved September 24, 2026, from https://learn.microsoft.com/en-us/fabric/mirroring/azure-databricks

Microsoft. (n.d.-j). Pause and resume your capacity. Microsoft Learn. Retrieved September 24, 2026, from https://learn.microsoft.com/en-us/fabric/enterprise/pause-resume

Microsoft. (n.d.-k). SQL analytics endpoint performance considerations. Microsoft Learn. Retrieved September 24, 2026, from https://learn.microsoft.com/en-us/fabric/data-warehouse/sql-analytics-endpoint-performance

Microsoft. (n.d.-l). SQL warehouse sizing, scaling, and queuing behavior. Microsoft Learn. Retrieved September 24, 2026, from https://learn.microsoft.com/en-us/azure/databricks/compute/sql-warehouse/warehouse-behavior

Microsoft. (n.d.-m). What is the SQL analytics endpoint for a lakehouse?. Microsoft Learn. Retrieved September 24, 2026, from https://learn.microsoft.com/en-us/fabric/data-engineering/lakehouse-sql-analytics-endpoint

Microsoft. (2026, August 12). Introducing Fabric Lakehouse support in dbt job for Microsoft Fabric (preview). Microsoft Fabric Community. https://community.fabric.microsoft.com/blog/fbc_fabricupdatesblogs/introducing-fabric-lakehouse-support-in-dbt-job-for-microsoft-fabric-preview/5358240

shashwotm. (2026, May 27). Fabric trial capacity - Too many requests for capacity error [Online forum post]. Microsoft Fabric Community. https://community.fabric.microsoft.com/t5/Data-Engineering/Fabric-Trial-Capacity-Too-Many-Requests-for-Capacity-Error/m-p/5189753